Large language model reasoning method, device, equipment, medium and program product

Abstract

The application relates to a large language model reasoning method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining corresponding KV data and an information matrix according to to-be-processed text through a first decoder in a large language model; determining a current decoder for this-time convergence judgment, obtaining KV data and an information matrix of the current decoder through the current decoder according to an information matrix of a previous decoder; determining mutual information between the information matrices of the previous decoder and the current decoder, taking a next decoder as a current decoder for next-time convergence judgment when the mutual information does not satisfy a convergence condition, continuously determining KV data and an information matrix of the current decoder for next-time convergence judgment until the convergence condition is satisfied; and determining to-be-reasoned data based on the KV data of the current decoder that satisfies the convergence condition, so that the to-be-reasoned data is used for model reasoning to obtain a reply text, and the large language model reasoning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for large speech model reasoning. Background Technology

[0002] With the development of natural language processing technology, large language model technology has emerged. By inputting text into a large language model, the large language model can perform corresponding reasoning based on the input text and generate corresponding response text.

[0003] In related technologies, after the large language model obtains the input text, before inference, it needs to go through a prefill stage to obtain KV (Key-Value) data for inference. That is, in the prefill stage, it needs to be processed by multiple pre-set decoders to obtain KV data, and then the model inference is performed based on the KV data to obtain the corresponding response text.

[0004] However, in related technologies, each input text needs to pass through each decoder in the pre-filling stage in sequence, resulting in a large amount of key-value data being generated each time. The amount of data for inference is large, making it difficult to ensure the inference efficiency of large language models. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the reasoning efficiency of large language models in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a large language model reasoning method, including:

[0007] After the text to be processed is fed into the large language model, the key-value data and information matrix corresponding to the first decoder are obtained through the first decoder in the large language model based on the text to be processed.

[0008] Determine the current decoder for the current convergence judgment, and obtain the KV data and information matrix corresponding to the current decoder through the information matrix of the previous decoder in the large language model.

[0009] The mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder is determined. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model is continued until the convergence condition is met.

[0010] The data to be inferred is determined based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0011] Secondly, this application also provides a large language model reasoning device, comprising:

[0012] The encoding processing module is used to obtain the KV data and information matrix corresponding to the first decoder in the large language model after the text to be processed is sent into the large language model.

[0013] The encoding processing module is also used to determine the current decoder for the current convergence judgment, and to obtain the KV data and information matrix corresponding to the current decoder through the current decoder based on the information matrix corresponding to the previous decoder in the large language model.

[0014] The convergence judgment module is used to determine the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model continues to be executed until the convergence condition is met.

[0015] The data acquisition module is used to determine the data to be inferred based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0016] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0017] After the text to be processed is fed into the large language model, the key-value data and information matrix corresponding to the first decoder are obtained through the first decoder in the large language model based on the text to be processed.

[0018] Determine the current decoder for the current convergence judgment, and obtain the KV data and information matrix corresponding to the current decoder through the information matrix of the previous decoder in the large language model.

[0019] The mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder is determined. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model is continued until the convergence condition is met.

[0020] The data to be inferred is determined based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0022] After the text to be processed is fed into the large language model, the key-value data and information matrix corresponding to the first decoder are obtained through the first decoder in the large language model based on the text to be processed.

[0023] Determine the current decoder for the current convergence judgment, and obtain the KV data and information matrix corresponding to the current decoder through the information matrix of the previous decoder in the large language model.

[0024] The mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder is determined. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model is continued until the convergence condition is met.

[0025] The data to be inferred is determined based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0026] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0027] After the text to be processed is fed into the large language model, the key-value data and information matrix corresponding to the first decoder are obtained through the first decoder in the large language model based on the text to be processed.

[0028] Determine the current decoder for the current convergence judgment, and obtain the KV data and information matrix corresponding to the current decoder through the information matrix of the previous decoder in the large language model.

[0029] The mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder is determined. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model is continued until the convergence condition is met.

[0030] The data to be inferred is determined based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0031] The aforementioned large language model inference method, apparatus, computer device, computer-readable storage medium, and computer program product, after feeding the text to be processed into the large language model, obtain the key-value data and information matrix corresponding to the first decoder in the large language model based on the text to be processed. Then, determine the current decoder for the current convergence judgment, and obtain the key-value data and information matrix corresponding to the current decoder based on the information matrix corresponding to the previous decoder in the large language model. Determine the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder. If the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment, and the process returns to the step of obtaining the key-value data and information matrix corresponding to the current decoder based on the information matrix corresponding to the previous decoder in the large language model, until the convergence condition is met. In other words, once the current decoder that meets the convergence condition has converged, the decoders after the current decoder in the large language model no longer need to be analyzed and can stop analysis, i.e., model inference can begin. In this way, the key-value (KV) data corresponding to the current decoder that meets the convergence condition is used to determine the data to be inferred. This data is then used for model inference to determine the response text to the text to be processed. Throughout the process, the mutual information between the current decoder and the previous decoder allows for timely determination of whether the current decoder has converged, enabling early cessation of subsequent decoder analysis. This avoids the need for all decoders to perform analysis, significantly reducing the amount of KV data required for model inference and thus greatly improving the inference efficiency of large language models. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a diagram illustrating the application environment of a large language model inference method in one embodiment.

[0034] Figure 2 This is a flowchart illustrating the large language model inference method in one embodiment;

[0035] Figure 3 This is a schematic diagram of the data packet format encapsulated by VxLan in one embodiment;

[0036] Figure 4This is a schematic diagram of the data packet format encapsulated by VxLan in another embodiment;

[0037] Figure 5 This is a schematic diagram of the control device in one embodiment;

[0038] Figure 6 This is a schematic diagram of the reasoning process in one embodiment;

[0039] Figure 7 This is a structural block diagram of a large language model inference device in one embodiment;

[0040] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0043] Before introducing this application, the following terms will be explained:

[0044] Large Language Model (LLM) is a deep learning model trained on a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text.

[0045] PD (Prefill and Decode phases) separation: In the reasoning process of large language models, the entire process is usually divided into two main phases: the Prefill phase and the Decode phase. Based on the operation of these two phases and their different resource requirements, they are deployed separately. This separation helps to improve reasoning efficiency and optimize the use of computing resources.

[0046] The large language model reasoning method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. In some embodiments, the large language model inference method provided in this application can be executed by terminal 102 or server 104 alone, or it can be executed collaboratively by terminal 102 and server 104; the specific execution method is not limited.

[0047] In some embodiments, after the terminal 102 obtains the text to be processed input by the user, it sends the text to the server 104. The server 104 sends the text to be processed into a large language model. After sending the text to be processed into the large language model, the server 104 obtains the key-value data and information matrix corresponding to the first decoder in the large language model based on the text to be processed. The server 104 determines the current decoder for the current convergence judgment, and obtains the key-value data and information matrix corresponding to the current decoder based on the information matrix corresponding to the previous decoder in the large language model. The server 104 determines the previous decoder. The mutual information between the information matrix corresponding to each decoder and the information matrix corresponding to the current decoder is considered. If the mutual information does not meet the convergence condition, the next decoder after the current decoder is used as the current decoder for the next convergence judgment. The process of obtaining the key-value data and information matrix corresponding to the current decoder based on the information matrix corresponding to the previous decoder in the large language model continues until the convergence condition is met. Server 104 determines the data to be inferred based on the key-value data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed. Server 104 returns the response text to terminal 102 so that terminal 102 can display the response text.

[0048] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0049] In one exemplary embodiment, such as Figure 2 As shown, a large language model reasoning method is provided, which can be applied to computer devices (which can be...). Figure 1 Taking terminal 102 (or server 104) as an example, the explanation includes the following steps 202 to 208. Wherein:

[0050] Step 202: After the text to be processed is fed into the large language model, the KV data and information matrix corresponding to the first decoder in the large language model are obtained based on the text to be processed.

[0051] In this context, the text to be processed is the text to be processed by the model inference process. In some embodiments, the text to be processed can be the text input by the current user. For example, the text to be processed is the query text input by the current user. The decoder in the large language model refers to the decoder involved in the prefilling stage. The decoder is used to extract features to obtain KV data (key-value data) and an information matrix representing the hidden state. In the subsequent decoding stage, model inference can be performed based on the KV data to obtain the response text, which is the text that replies to the text to be processed. KV data can also be understood as KVcache data, used for model inference. The information matrix is ​​a vector matrix representing the hidden state. In some embodiments, the large language model is a Transformer (sequence converter) large language model. In some embodiments, the decoder is the core component of the large language model. In the prefill stage of the large language model, the user input text undergoes deep analysis after passing through multiple layers of decoders. In some embodiments, the core components of the decoder are the self-attention layer and the feedforward neural feedback layer. Large language models include multiple decoders cascaded, such as 96 or 120. The input text is processed by the self-attention layer and feedforward network layer of each decoder to extract features and perform semantic analysis. The self-attention layer outputs an attention weight matrix after analysis, with each row representing a token. Shallow self-attention layers perform contextual analysis on the input text's vector matrix, extracting relationships and features between different words. Deeper self-attention layers perform more complex semantic analysis, capturing long-range dependencies and outputting the text's attention weight matrix. The output of the self-attention layer is then processed through residual connections and layer normalization before being fed to the feedforward neural network layer, which performs feature enhancement through non-linear transformations.

[0052] Optionally, after acquiring the text to be processed, the computer device sends the text to a large language model, which then performs word segmentation. The segmentation results are input into the first decoder of the large language model, which extracts features from the segmented results. The first decoder outputs the corresponding key-value data and an information matrix. Each row of the information matrix corresponds to a token, and each row represents the feature vector of the corresponding token extracted by the decoder. The feature vector is multi-dimensional; therefore, the information matrix can be understood as having multiple rows and columns. For each element in the information matrix, the element represents the feature value of the token in the corresponding column dimension.

[0053] Step 204: Determine the current decoder for the current convergence judgment. Based on the information matrix of the previous decoder in the large language model, obtain the KV data and information matrix corresponding to the current decoder through the current decoder.

[0054] In this context, the current decoder is not the first decoder. The convergence check determines whether the information matrix of the decoders converges. Therefore, it can be understood that the current convergence check is to determine whether the information matrix of the current decoder converges. The previous decoder before the current decoder refers to the decoder that is adjacent to and precedes the current decoder. For example, in a large language model with decoders 1 to 90, and the current decoder is decoder 12, then the previous decoder is decoder 11. Of course, the previous decoder can also be understood as the current decoder in the previous convergence check.

[0055] Optionally, starting from a non-first decoder, when the computer device determines the current decoder for the current convergence judgment, the computer device obtains the information matrix corresponding to the previous decoder located before the current decoder, inputs the information matrix corresponding to the previous decoder into the current decoder, processes the information matrix through the current decoder, and outputs the corresponding KV data and information matrix.

[0056] For example, when the current decoder in the convergence judgment is decoder i (i is a positive integer not equal to 1), the computer device determines the information matrix corresponding to decoder i-1. ,Will The input is fed into decoder i, and the output is the KV data (datai) and information matrix corresponding to decoder i. .

[0057] Step 206: Determine the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder. If the mutual information does not meet the convergence condition, take the next decoder after the current decoder as the current decoder for the next convergence judgment, and continue to execute the step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model, until the convergence condition is met.

[0058] Mutual information is used to reflect the redundancy between two corresponding information matrices. For example, the smaller the mutual information between any two information matrices, the less redundancy exists between them. A mutual information value of 0 indicates that the two information matrices have no redundancy. In some embodiments, for two adjacent decoders, the larger the mutual information of their respective information matrices, the closer they are to convergence. For example, for the current decoder, the larger the mutual information of the information matrices of the previous decoder and the current decoder, the more convergent the pre-filling process in the current pre-filling stage is, and the less meaningful it is for subsequent decoders (decoders after the current decoder) to process it; that is, the pre-filling stage can be considered complete. The convergence condition is used to determine whether a next convergence judgment is needed, i.e., whether the next decoder after the current decoder should process it. In some embodiments, the convergence condition is that the mutual information is greater than or equal to an information threshold; or, in some embodiments, the convergence condition is that the mutual information is greater than or equal to an information threshold, and the current convergence count reaches the first threshold. Alternatively, in some embodiments, the total number of convergence judgments where the mutual information is greater than or equal to the information threshold is counted. The convergence condition is that the mutual information is greater than or equal to the information threshold and the total number of judgments reaches the second threshold.

[0059] Optionally, the computer device calculates the probability distribution based on the information matrix corresponding to the previous decoder to obtain the probability distribution corresponding to the previous decoder. The computer device also calculates the probability distribution based on the information matrix corresponding to the current decoder to obtain the probability distribution corresponding to the current decoder. Based on the probability distributions of both the previous and current decoders, the computer device obtains the mutual information of the current decoder, thus determining the mutual information between the information matrices of the previous and current decoders.

[0060] Optionally, the computer device determines whether the convergence condition is met based on mutual information. If the convergence condition is met, the convergence determination stops. If the convergence condition is not met, the computer device determines the next decoder that is adjacent to the current decoder from multiple consecutive decoders in the large language model. This next decoder is used as the current decoder for the next convergence determination. The process of obtaining the KV data and information matrix corresponding to the current decoder based on the information matrix of the previous decoder in the large language model is continued until the convergence condition is met. For example, if the current decoder for the current convergence determination is decoder i, and the convergence condition is not met, decoder i+1 is used as the current decoder for the next convergence determination.

[0061] Step 208: Determine the data to be inferred based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0062] In this process, model inference is performed during the decoding phase, which involves reasoning from the key-value data obtained in the pre-filling phase to arrive at the response text. The response text can be understood as the answer to the text to be processed.

[0063] It should be noted that steps 202 to 208 above are all in the pre-filling stage of the large language model. In some embodiments, the pre-filling stage and the decoding stage of the large language model can be processed by different devices. For distinction, the implementing entity of the decoding stage is referred to as another device, that is, a different device from the computer device implementing the pre-filling stage. This other device can be a terminal or a server.

[0064] It should be noted that by determining whether the mutual information satisfies the convergence condition in the above process, not only can the inference efficiency be improved, but also the subsequent decoder that has met the convergence condition should not continue to process even if convergence has been achieved, which can greatly save computing resources.

[0065] In the aforementioned large language model inference method, after the text to be processed is fed into the large language model, the first decoder in the large language model obtains the key-value data and information matrix corresponding to the first decoder based on the text. Then, the current decoder for the current convergence judgment is determined. Based on the information matrix of the previous decoder in the large language model, the key-value data and information matrix corresponding to the current decoder are obtained through the current decoder. The mutual information between the information matrix of the previous decoder and the information matrix of the current decoder are determined. If the mutual information does not meet the convergence condition, the next decoder after the current decoder is used as the current decoder for the next convergence judgment. The process returns to the previous step of obtaining the key-value data and information matrix of the current decoder based on the information matrix of the previous decoder in the large language model, until the convergence condition is met. In other words, once the current decoder that meets the convergence condition has converged, the decoders after the current decoder in the large language model no longer need to be analyzed and can stop analysis, allowing model inference to proceed. In this way, the key-value (KV) data corresponding to the current decoder that meets the convergence condition is used to determine the data to be inferred. This data is then used for model inference to determine the response text to the text to be processed. Throughout the process, the mutual information between the current decoder and the previous decoder allows for timely determination of whether the current decoder has converged, enabling early cessation of subsequent decoder analysis. This avoids the need for all decoders to perform analysis, significantly reducing the amount of KV data required for model inference and thus greatly improving the inference efficiency of large language models.

[0066] In some embodiments, determining the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder includes: determining the joint probability distribution corresponding to the current decoder based on the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder; determining the marginal probability distribution corresponding to the previous decoder based on the information matrix corresponding to the previous decoder, and determining the marginal probability distribution corresponding to the current decoder based on the information matrix corresponding to the current decoder; and determining the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder based on the marginal probability distributions corresponding to the previous decoder and the current decoder, and the joint probability distribution corresponding to the current decoder.

[0067] For example, for ease of understanding, the information matrix corresponding to the current decoder is regarded as the first information matrix, and the information matrix corresponding to the previous decoder is regarded as the second information matrix. It should be noted that the information matrix obtained by each decoder has the same size, and the number of rows and columns of the information matrices obtained by any two decoders are the same. For each first element value in the first information matrix, the position of the first element value in the first information matrix is ​​determined (i.e., the row number and column number of the first element). Based on the row number and column number, the second element value located at the row number and column number is obtained from the second information matrix. The product of the first element value and the second element value is taken as the product value corresponding to the first element value, and the product values ​​corresponding to each first element value are superimposed to obtain the first sum value. For each first element value, the ratio of the product value corresponding to the first element value to the first sum value is taken as the joint probability distribution corresponding to the position. For example, for the position (t, j) in the t-th row and j-th column, the joint probability distribution of the position is calculated using the following formula (1). :

[0068] (1)

[0069] in, This is the information matrix corresponding to the previous decoder (decoder i-1), i.e., the second information matrix, where X... i-1 R n×d . Let X be the information matrix corresponding to the current decoder (decoder i), where X i R n×d That is, the first information matrix. This represents the value of the second element at position (t, j) in the second information matrix. Let be the value of the first element at position (t, j) in the first information matrix. n is the number of rows, and d is the number of columns. The values ​​of the joint probability distribution are between 0 and 1.

[0070] For example, for any information matrix in the first information matrix and the second information matrix, the probability distribution of the corresponding position can be determined using the following formula (2) based on any position in any information matrix:

[0071] (2)

[0072] in, Let be the probability distribution of position (t, j) in any information matrix. Let be the element value at position (t, j) in any information matrix.

[0073] Next, the marginal probability distribution at any position in any information matrix can be determined using the following formula (3):

[0074] (3)

[0075] in, Marginal probability distribution of position (t, j) in any information matrix.

[0076] Finally, for any information matrix in the first and second information matrices, after obtaining the marginal probability distribution of each position in any information matrix using the above formula, the marginal probability distribution corresponding to any information matrix is ​​obtained based on the probability distribution of each position in any information matrix. Then, the marginal probability distribution corresponding to the first information matrix and the marginal probability distribution corresponding to the second information matrix are multiplied together, that is, the probability distributions at the same position are multiplied together to obtain the first operation result (including the product value of each position). The joint probability distribution corresponding to the current decoder (that is, the joint probability distribution of each position obtained by formula (1)) and the first operation result are divided by the product value, that is, the joint probability distribution at the same position is divided by the product value to obtain the second operation result. The second operation result is logarithmically calculated to obtain the third operation result (including the logarithmic value of each position). The joint probability distribution corresponding to the current decoder and the third operation result are multiplied together, that is, the joint probability distribution at the same position is multiplied by the logarithmic value to obtain the fourth operation result, that is, the mutual information is obtained.

[0077] Specifically, for any information matrix in the first and second information matrices, after obtaining the joint probability distribution at any position and the marginal probability distribution of any information matrix, flatten any n×d two-dimensional information matrix into a one-dimensional vector of length n×d, and use the following formula (4) to calculate the mutual information between the information matrix corresponding to the previous decoder i-1 and the information matrix corresponding to the current decoder i. :

[0078] (4)

[0079] Furthermore, the value of mutual information is greater than or equal to 0 and is symmetric. When it is equal to 0, it means that there is no redundancy between the two vector matrices (this situation is very unlikely in large models), while the upper limit can be infinitely large. Therefore, normalization should be performed. Since the mutual information value is related to information, it can be calculated using the information entropy of the information matrix, as shown in the following formula (5):

[0080] (5)

[0081] in, Let be the information entropy of any information matrix. This refers to the marginal probability distribution corresponding to any information matrix mentioned above (i.e., the marginal probability distribution of each position in any information matrix).

[0082] Then, the normalized mutual information δ is obtained using formula (6):

[0083] δ = Norm(I(X) i ,X i+1 ))=( 2 × I(X i ,X i+1 )) / (H(X i ) + H(X i+1 (6)

[0084] Then, it is determined whether the normalized mutual information satisfies the convergence condition. If it does not, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix of the previous decoder in the large language model is returned and the process continues until the convergence condition is met.

[0085] It should be noted that, through normalization, the mutual information value of the vector matrix outputs of two adjacent decoders is calculated. It is between 0 and 1. The closer it is to 1, the greater the mutual information is, and it approaches the convergence state. When the mutual information value of the vector matrix outputs of multiple consecutive decoders is very high, it can be considered that the current pre-filling process is in a convergence state. Subsequent decoder processing is not very meaningful, and the current round of pre-filling process is stopped.

[0086] In this embodiment, based on the marginal probability distributions corresponding to the previous decoder and the current decoder, and the joint probability distribution corresponding to the current decoder, the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder can be accurately evaluated. This allows for accurate determination of whether the convergence condition is met, ensuring the effectiveness and accuracy of large language model inference.

[0087] In some embodiments, the method further includes: obtaining an information content threshold; comparing mutual information content and the information content threshold to obtain a comparison result; and determining whether the mutual information content satisfies the convergence condition based on the comparison result.

[0088] The information content threshold is a pre-set value that determines whether the pre-filling stage has begun to converge. In some embodiments, the information content threshold can be determined based on historical inference data. For example, the text complexity corresponding to the input text can be determined (which can be based on the number of words; the more words, the more complex). Historical input texts with similar text complexity are obtained, and the information content thresholds corresponding to each of the historical input texts are obtained. The average of the information content thresholds corresponding to each of the historical input texts is then used as the information content threshold for the input text.

[0089] For example, if the mutual information is less than the information threshold, it is determined that the mutual information does not meet the convergence condition. If the mutual information is greater than or equal to the information threshold, it is determined that the mutual information meets the convergence condition.

[0090] In this embodiment, by comparing the mutual information and the information threshold, it is possible to quickly determine whether the convergence condition is met, thereby improving the reasoning efficiency of the large language model.

[0091] In some embodiments, determining whether the mutual information content satisfies the convergence condition based on the comparison results includes: if the mutual information content is less than the information content threshold, determining that the mutual information content does not satisfy the convergence condition; if the mutual information content is greater than or equal to the information content threshold, obtaining the previous convergence quantity corresponding to the previous data update, and adding a unit value to the previous convergence quantity to obtain the current convergence quantity corresponding to the current data update; if the current convergence quantity is less than the quantity threshold, determining that the mutual information content does not satisfy the convergence condition; and if the current convergence quantity is equal to the quantity threshold, determining that the mutual information content satisfies the convergence condition.

[0092] The previous data update begins after the mutual information of the second decoder is determined. Data updates occur when the mutual information is greater than or equal to an information threshold. The previous data update is performed when the mutual information of the preceding decoder is determined to be greater than or equal to the information threshold during the comparison of the mutual information of the preceding decoder with the information threshold. The preceding decoder is the decoder before the current decoder; it could be the previous decoder, the two previous decoders, the three previous decoders, etc. The current decoder is decoder i, and the number of previous convergences corresponding to the previous data update is less than or equal to i-2.

[0093] For example, if the mutual information is less than the information threshold, the computer device determines that the mutual information does not meet the convergence condition; if the mutual information is greater than or equal to the information threshold, the computer device obtains the previous convergence quantity corresponding to the previous data update from the counter, increments the previous convergence quantity by 1, and obtains the current convergence quantity corresponding to the current data update; if the current convergence quantity is less than the quantity threshold, it determines that the mutual information does not meet the convergence condition; if the current convergence quantity is equal to the quantity threshold, it determines that the mutual information meets the convergence condition. For instance, if the quantity threshold is 10, the current decoder is decoder 6, and the previous convergence quantity is 3, if the mutual information corresponding to decoder 6 (calculated based on the information matrices of decoders 5 and 6) is greater than the information threshold, then the current convergence quantity is 4. Since the current convergence quantity is less than 10, it is determined that the convergence condition has not been met.

[0094] In this embodiment, the ratio of mutual information to the information threshold and the ratio of the current number of converged units to the number threshold are used to comprehensively determine whether the mutual information of the current decoder meets the convergence condition, that is, to determine whether the convergence is complete. This can accurately identify the convergence timing, avoid unnecessary processing of subsequent decoders, save resources, simplify the inference data used for inference, and improve the inference efficiency of large language models.

[0095] In some embodiments, the step of determining the number of previous convergences includes: obtaining at least one target decoder from the preceding decoders before the current decoder, wherein each target decoder is a preceding decoder whose mutual information is greater than or equal to the information threshold; and using the number of target decoders as the number of previous convergences.

[0096] The number of target decoders can be 0, meaning that none of the preceding decoders has mutual information greater than or equal to the information threshold, or it can be 1, or a positive integer such as 2.

[0097] For example, at least one undetermined decoder is obtained from the preceding decoders before the current decoder, where the undetermined decoder is the preceding decoder whose mutual information is greater than or equal to the information threshold.

[0098] Furthermore, the number of previous convergences is determined based on the following cases: Case 1: When the number of undetermined decoders is 0, the number of target decoders is 0, and the number of previous convergences is 0. Case 2.1: If the number of undetermined decoders is 1, and the undetermined decoder is not the previous decoder of the current decoder, then the number of target decoders is 0, i.e., the number of previous convergences is 0; Case 2.2: If the number of undetermined decoders is 1, and the undetermined decoder is the previous decoder, then the target decoder is the previous decoder, and the number of previous convergences is 1.

[0099] Case 3.1: If the number of undetermined decoders is greater than 1, and there is no previous decoder among the undetermined decoders, then the number of target decoders is 0, that is, the number of previous convergences is 0. Case 3.2: If the number of undetermined decoders is greater than 1, and there is a previous decoder among the undetermined decoders, then check if there are consecutive decoders among the undetermined decoders. The decoder with the largest sequence number among the consecutive decoders is the previous decoder. If there are consecutive decoders, then all consecutive decoders are used as target decoders. If there are no consecutive decoders, then the previous decoder is the target decoder.

[0100] For example: The current decoder is decoder 6. The preceding decoders are decoder 1 to decoder 5. If the mutual information of decoder 3 (calculated based on the information matrices of decoder 2 and decoder 3), the mutual information of decoder 4 (calculated based on the information matrices of decoder 3 and decoder 4), and the mutual information of decoder 5 (calculated based on the information matrices of decoder 4 and decoder 5) are all greater than the information threshold, and the mutual information of decoder 2 (calculated based on the information matrices of decoder 1 and decoder 2) is less than the information threshold, then decoder 3 to decoder 5 are all target decoders, and the number of previous convergences is 3.

[0101] If the mutual information of decoder 2, decoder 3, and decoder 4 are all less than the information threshold, and the mutual information of decoder 5 is greater than the information threshold, then decoder 5 is the target decoder, and the previous convergence number is 1.

[0102] At this point, if the mutual information of decoder 4 and decoder 5 are both greater than the information threshold, and the mutual information of decoder 2 and decoder 3 is less than the information threshold, then decoder 4 and decoder 5 are the target decoders, and the number of previous convergences is 2.

[0103] In this embodiment, at least one target decoder is obtained from the preceding decoders before the current decoder. Each target decoder is a preceding decoder whose mutual information is greater than or equal to the information threshold. Therefore, the number of target decoders is used as the number of previous convergences. In this way, the accurate number of previous convergences can be counted, ensuring the accuracy of the convergence judgment.

[0104] In some embodiments, determining the data to be inferred based on the KV data corresponding to the current decoder that satisfies the convergence condition includes: taking the current decoder that satisfies the convergence condition as the convergent decoder, determining the intermediate decoders located between the first decoder and the convergent decoder; and obtaining the data to be inferred based on the KV data corresponding to the first decoder, the KV data corresponding to the convergent decoder, and the KV data corresponding to each intermediate decoder.

[0105] For example, the key-value data corresponding to the first decoder, the key-value data corresponding to the convergent decoder, and the key-value data corresponding to each intermediate decoder are aggregated to obtain the data to be inferred.

[0106] For example, the intermediate decoder may have a corresponding decoder with mutual information less than the information threshold, or it may not have a corresponding decoder with mutual information less than the information threshold.

[0107] For example, after determining the data to be inferred, it needs to be sent to the inference node, i.e., the device that performs model inference during the decoding phase. The inference node can be encapsulated using the VxLAN (Virtual Extended Local Area Network) packet format to send the encapsulated data to it. For example... Figure 3 The diagram shows a schematic of the data packet format encapsulated by VxLan in one embodiment. The VxLan header is 8 bytes long. The first byte is a preset byte, indicating that it is a VxLan message. The second to fourth bytes are reserved bytes, and the fifth to seventh bytes are the VNI field, which is the ID value of the VxLan tunnel. This VNI value is a preset ID number of the computer device and the inference node. The last byte is also a reserved byte.

[0108] Furthermore, it is typically necessary to send the data to be inferred to the inference node in batches. The sub-data sent in each batch can be encrypted as needed. To ensure that the inference node knows the details of each sent sub-data (such as whether it is encrypted and which part of the data to be processed is being transmitted) when receiving sub-data, a VxLAN encapsulation method can be used. For example, the first reserved field, i.e., bytes 2 to 4, can be used. Specifically, for the second byte: the first bit indicates whether the data packet is encrypted; encryption is set to 1, and no encryption is set to 0. The last 7 bits are the encoding bits for the encryption method. This invention does not involve data encryption, therefore the encoding of the encryption algorithm is not defined in detail. For example, a field value of 000000 indicates that the KV data encapsulated in this data packet is unencrypted; a field value of 100001 indicates that the KV data has been encrypted using the AES algorithm, etc. Regarding the processing of byte 3: This byte indicates the total number of layers in the transmitted KV data (the total number of layers can be determined based on the decoder sequence number of the convergent decoder; for example, if the sequence number of the convergent decoder is 50, it means that convergence is complete when the 50th decoder is reached, and the total number is 50). Therefore, one byte is sufficient to express the total number of layers in the KV data. That is, when processing reaches this layer, convergence is determined by calculating the mutual information redundancy of the vector matrix, and no further processing through other layers is required. Regarding the processing of byte 4: This byte indicates the layer sequence number of the KV data transmitted in the current data packet. The value of this field is less than the value of byte 3. For example... Figure 4The diagram shown illustrates the data packet format encapsulated by VxLan in another embodiment. When encapsulating any sub-data within the data to be inferred, a method such as... Figure 4 The data packet is encapsulated using the encapsulation method to obtain a data packet. This data packet indicates that the 2nd, 3rd, and 4th bytes of the VxLan header indicate that the current KV Cache data (sub-data) is encrypted using the AES encryption algorithm. There are a total of 61 layers of KV Cache data, and the sub-data being transmitted is the 36th layer of KV Cache data.

[0109] In the above embodiments, the current decoder that meets the convergence condition is taken as the converged decoder, indicating that the decoding layer of the pre-filled node only needs to process up to the converged decoder. The self-attention layer of the decoder after the converged decoder has little effect and does not need to be further processed by the decoder after the converged decoder. At this time, the KV data of all decoders before the converged decoder and the KV data of the converged decoder are taken as the data to be inferred, so that efficient model inference can be performed based on comprehensive but optimized data to be processed.

[0110] In one specific embodiment, a control device is involved in the pre-filling inference stage. This control device is used to execute the large language model inference method provided in the embodiments of this application, such as... Figure 5 The diagram shown is a schematic representation of a control device in one embodiment. This control device is a computer device, which may be a terminal or a server. (Refer to...) Figure 5 Key components of the control device include: a control unit for the pre-filling stage, a large language model KV data storage, an information matrix mutual information calculation engine, a VxLan data packet encapsulation engine, a large language model decoder array, an input text receiver, an m-counter, and an s-counter. Value storage Value Memory. Let's first explain these key components: Pre-filling Stage Control Unit: This control unit is the core unit of adaptive scheduling. Based on relevant preset parameters, it is responsible for scheduling inference convergence and KV data scheduling during the pre-filling stage. During the pre-filling stage, after each decoder completes the processing of the information matrix, the control unit forwards the output information matrix to the next decoder and simultaneously outputs it to the information matrix mutual information calculation engine to calculate the mutual information of the vector matrices between decoders. Large Language Model KV Data Memory: A high-speed cache for the large language model KV data, corresponding to the large model decoder. Information Matrix Mutual Information Calculation Engine: Under the control of the adaptive scheduler, this calculation engine calculates the mutual information of the information matrices output by two adjacent decoders. This includes the calculation algorithm, calculation completion, and sending the calculation results to the pre-filling stage control unit. VxLAN Data Packet Encapsulation Engine: This encapsulation engine, also under the control of the adaptive scheduler, extracts the KV data of the multi-layer decoders from the large language model KV data memory, encapsulates it into VxLAN data packets according to rules, and sends them to the inference node (Decode computation node). Large Language Model Decoder Array: Contains decoders of a multi-layer Transformer large language model. Each decoder layer includes a self-attention layer, a feedforward layer, and residual connections and layer normalization layers. The output information matrix of each layer serves as the input to the next layer's decoder. Input Text (Inference Text) Receiver: Receives the user's inference task and the input text for inference, and performs the inference process under the control of the scheduler. m-Counter: This counter records whether the cosine similarity (i.e., mutual information) of the output information matrices of any two adjacent decoders exceeds a preset threshold. (i.e., information threshold), once it occurs, it will step by 1 under the control of the scheduler, with a minimum of 2. If it is found that it has not been exceeded in any stage, it will be immediately set to 1. s counter: This counter records the decoder sequence number of the currently processed input information matrix in the pre-filling stage. Once the scheduler finds convergence, it will adaptively stop inference, and the KV data pushed to the inference node will be from sequence number 1 to s layer. Value storage: It is a preset value, stored in this memory. Value storage: It is a preset value (the quantity threshold mentioned above) stored in this unit.

[0111] In view of this, the specific reasoning process of the large language model in this application will be explained below, such as... Figure 6 The diagram shown is a schematic representation of the reasoning process in one embodiment. Figure 6 middle, This is the information content threshold, which is also the mutual information convergence threshold of the information matrix; The threshold is the number of consecutive mutual information in the decoder layer, which is greater than or equal to the threshold value. The minimum value for T is 2; the total number of layers in the large language model decoder is 2; s is a counter, starting from 1, counting the decoder layer number in the current pre-filling stage; m is a counter that records the current consecutive mutual information value greater than 2. The number of decoder layers. Here, the number of decoder layers can be understood as the number of decoders.

[0112] The specific steps are as follows:

[0113] Step 1: After the input text receiver receives the inference request and the input text, it starts the pre-filling stage. At this time, the counters are set, i.e., s=1; m=1. Then, Step 2 is executed.

[0114] Step 2: In the pre-filling stage, the control unit sends the input text to the token segmenter and embedding layer to generate a token vector matrix, and then sends it to the first decoder to obtain the KV data and information matrix corresponding to the first decoder.

[0115] Step 3: For the current decoder that is not the first decoder, the control unit in the pre-filling stage determines whether the sequence number of the current decoder is the last layer of the large language model. If not, proceed to step 4. If yes, exit the pre-filling stage and proceed to step 7.

[0116] Step 4: In the last layer of the large model, i.e., s is less than T. When s is greater than 1, the control unit sends the information matrix of the previous decoder to the information matrix mutual information calculation engine. The information matrix mutual information calculation engine calculates the normalized mutual vector between the information matrices of the current decoder and the previous decoder. Then, proceed to step 5.

[0117] Specifically, the mutual information calculation engine determines the joint probability distribution of the current decoder based on the information matrices of the previous and current decoders. It also determines the marginal probability distribution of the previous decoder based on its own information matrix, and further determines the marginal probability distribution of the current decoder based on its own information matrix. Based on the normalized marginal probability distributions of both the previous and current decoders, and the joint probability distribution of the current decoder, the mutual information between the information matrices of the previous and current decoders is determined. Finally, based on the marginal probability distributions of both the current and previous decoders, the information entropy of their respective information matrices is calculated. The mutual information is then normalized based on these entropies to obtain the normalized mutual information.

[0118] Step 5: The mutual information calculation engine of the information matrix determines whether the normalized mutual information is ≥ If not, proceed to step 6 below. If yes, increment m in the m-value counter by 1, and determine if m is equal to φ. If it is equal, it indicates that the current pre-filling stage has reached the convergence condition. Save the current level number s, exit the pre-filling stage, and proceed to step 7. Step 6: m=1, set m to 1, and then s++, that is, update by incrementing s by 1, continue to the next layer of decoder, and return to step 3 to continue execution.

[0119] Step 7: After the pre-filling phase ends, based on the recorded decoder sequence number s of the current large model, the KV data generated by the first layer decoder to the s-th layer decoder are layered and encapsulated into data packets, and the data is sent to the inference node. The inference node receives the inference task and the KV data, completes the Decode Token generation, and outputs the response text.

[0120] It should be noted that the large language model inference method provided in this application employs a triggering mechanism, which can monitor the convergence of the perception pre-filling stage in a relatively simple and effective way. For inference with low complexity, it can save the computing resources of the control device and also significantly reduce the amount of KV data that needs to be transmitted to the inference node. Furthermore, it is adaptive control, without the need to introduce too many parameters and decision algorithms.

[0121] It should be noted that different large language model operators can flexibly pre-configure according to the actual needs of their inference business. and , It can be set between 3 and 5, and It can be set between 0.8 and 0.95. It is important to emphasize that the low complexity of the pre-filling stage does not necessarily mean that the decoding stage will be simple. For example, if the input text or prompt is "Please explain relativity in detail", the current large language model will be relatively simple in the pre-filling stage if it does not need to perform network search, and it can converge with only a few decoder layers. However, the task output by the inference node in the decoding stage will be very complex, and it will output a lot of content based on the knowledge learned during training. Therefore, the computation of the pre-filling stage is not affected by the convergence scheduling of the preprocessing stage.

[0122] It should be noted that the core idea of ​​the reasoning process of the above-mentioned large language model is to analyze and calculate the redundancy of mutual information of vector matrices in the pre-filling stage, dynamically select the convergence time of the pre-filling stage, and reduce the overall volume of KV data transmission, thus significantly reducing the magnitude of KV data transmission.

[0123] In this embodiment, after the text to be processed is fed into the large language model, the first decoder in the large language model obtains the key-value data and information matrix corresponding to the first decoder based on the text to be processed. Then, the current decoder for the current convergence judgment is determined. Based on the information matrix corresponding to the previous decoder in the large language model, the key-value data and information matrix corresponding to the current decoder are obtained through the current decoder. The mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder are determined. If the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The process returns to the step of obtaining the key-value data and information matrix corresponding to the current decoder based on the information matrix corresponding to the previous decoder in the large language model, until the convergence condition is met. That is, once the current decoder that meets the convergence condition has converged, the decoders after the current decoder in the large language model no longer need to be analyzed and can stop analysis, i.e., model inference can be performed. In this way, the key-value (KV) data corresponding to the current decoder that meets the convergence condition is used to determine the data to be inferred. This data is then used for model inference to determine the response text to the text to be processed. Throughout the process, the mutual information between the current decoder and the previous decoder allows for timely determination of whether the current decoder has converged, enabling early cessation of subsequent decoder analysis. This avoids all decoders performing analysis, significantly reducing the amount of KV data used for model inference. While accelerating the pre-filling stage, it also significantly reduces the amount of KV data transmitted by inference nodes, especially for inference services with relatively low complexity, thereby greatly improving the inference efficiency of large language models. Furthermore, it saves computational resources while significantly reducing the pressure of massive KV cache data transmission, thus achieving adaptive processing of KV data based on the complexity of the input text and reduced caching of KV data. Moreover, since it is implemented adaptively through preset parameters, the algorithm is not complex, possessing universality, versatility, and cross-platform compatibility. It can be implemented and deployed on various platform devices, as well as on physical devices or virtual machines, demonstrating broad application prospects.

[0124] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0125] Based on the same inventive concept, this application also provides a large language model reasoning apparatus for implementing the large language model reasoning method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the large language model reasoning apparatus provided below can be found in the limitations of the large language model reasoning method described above, and will not be repeated here.

[0126] In one exemplary embodiment, such as Figure 7 As shown, a large language model inference device 700 is provided, including: an encoding processing module 702, a convergence judgment module 704, and a data acquisition module 706, wherein:

[0127] The encoding processing module 702 is used to, after sending the text to be processed into the large language model, obtain the KV data and information matrix corresponding to the first decoder in the large language model based on the text to be processed;

[0128] The encoding processing module 702 is also used to determine the current decoder for the current convergence judgment, and to obtain the KV data and information matrix corresponding to the current decoder through the current decoder based on the information matrix corresponding to the previous decoder in the large language model.

[0129] The convergence judgment module 704 is used to determine the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model continues to be executed until the convergence condition is met.

[0130] The data acquisition module 706 is used to determine the data to be inferred based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

[0131] In some embodiments, the convergence determination module 704 is configured to determine the joint probability distribution of the current decoder based on the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder; determine the edge probability distribution of the previous decoder based on the information matrix corresponding to the previous decoder; determine the edge probability distribution of the current decoder based on the information matrix corresponding to the current decoder; and determine the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder based on the edge probability distributions corresponding to the previous decoder and the current decoder, and the joint probability distribution corresponding to the current decoder.

[0132] In some embodiments, the convergence determination module 704 is further configured to obtain an information content threshold; compare the mutual information content with the information content threshold to obtain a comparison result; and determine whether the mutual information content satisfies the convergence condition based on the comparison result.

[0133] In some embodiments, the convergence determination module 704 is further configured to: determine that the mutual information content does not meet the convergence condition when the mutual information content is less than the information content threshold; obtain the previous convergence quantity corresponding to the previous data update when the mutual information content is greater than or equal to the information content threshold, and obtain the current convergence quantity corresponding to the current data update by superimposing the previous convergence quantity with a unit value; determine that the mutual information content does not meet the convergence condition when the current convergence quantity is less than the quantity threshold; and determine that the mutual information content meets the convergence condition when the current convergence quantity is equal to the quantity threshold.

[0134] In some embodiments, the apparatus further includes a quantity determination module for obtaining at least one target decoder from the preceding decoders before the current decoder, wherein each target decoder is a preceding decoder whose mutual information is greater than or equal to an information threshold; and the number of target decoders is used as the number of previous convergences.

[0135] In some embodiments, the data acquisition module 706 is used to take the current decoder that meets the convergence condition as the convergent decoder, determine the intermediate decoder located between the first decoder and the convergent decoder, and obtain the data to be inferred based on the KV data corresponding to the first decoder, the KV data corresponding to the convergent decoder, and the KV data corresponding to each intermediate decoder.

[0136] Each module in the aforementioned large language model inference device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0137] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a large language model reasoning method.

[0138] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0141] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A reasoning method for large language models, characterized in that, The method includes: After the text to be processed is fed into the large language model, the key-value data and information matrix corresponding to the first decoder are obtained through the first decoder in the large language model based on the text to be processed. Determine the current decoder for the current convergence judgment, and obtain the KV data and information matrix corresponding to the current decoder through the information matrix of the previous decoder in the large language model. The mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder is determined. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model is continued until the convergence condition is met. The data to be inferred is determined based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

2. The method according to claim 1, characterized in that, Determining the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder includes: Based on the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder, determine the joint probability distribution corresponding to the current decoder; Based on the information matrix corresponding to the previous decoder, the edge probability distribution corresponding to the previous decoder is determined; based on the information matrix corresponding to the current decoder, the edge probability distribution corresponding to the current decoder is determined. Based on the edge probability distributions corresponding to the previous decoder and the current decoder, and the joint probability distribution corresponding to the current decoder, the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder is determined.

3. The method according to claim 1, characterized in that, The method further includes: Information acquisition threshold; The mutual information and the information threshold are compared to obtain the comparison result; Based on the comparison results, it is determined whether the mutual information satisfies the convergence condition.

4. The method according to claim 3, characterized in that, Determining whether the mutual information satisfies the convergence condition based on the comparison result includes: If the mutual information is less than the information threshold, it is determined that the mutual information does not meet the convergence condition. When the mutual information is greater than or equal to the information threshold, the previous convergence quantity corresponding to the previous data update is obtained, and the previous convergence quantity is superimposed with a unit value to obtain the current convergence quantity corresponding to the current data update. If the number of converged instances is less than the threshold value, it is determined that the mutual information does not meet the convergence condition. If the number of converged quantities in the current iteration is equal to the quantity threshold, it is determined that the mutual information quantity satisfies the convergence condition.

5. The method according to claim 4, characterized in that, The step of determining the number of previous convergences includes: Obtain at least one target decoder from the preceding decoders before the current decoder, where each target decoder is a preceding decoder whose mutual information is greater than or equal to the information threshold. The number of target decoders is used as the number of previous convergences.

6. The method according to claim 1, characterized in that, The process of determining the data to be inferred based on the KV data corresponding to the current decoder that meets the convergence condition includes: The current decoder that meets the convergence condition is taken as the convergent decoder, and the intermediate decoder located between the first decoder and the convergent decoder is determined. Based on the KV data corresponding to the first decoder, the KV data corresponding to the convergent decoder, and the KV data corresponding to each intermediate decoder, the data to be inferred is obtained.

7. A large language model reasoning device, characterized in that, The device includes: The encoding processing module is used to obtain the KV data and information matrix corresponding to the first decoder in the large language model after the text to be processed is sent into the large language model. The encoding processing module is also used to determine the current decoder for the current convergence judgment, and to obtain the KV data and information matrix corresponding to the current decoder through the current decoder based on the information matrix corresponding to the previous decoder in the large language model. The convergence judgment module is used to determine the mutual information between the information matrix corresponding to the previous decoder and the information matrix corresponding to the current decoder. When the mutual information does not meet the convergence condition, the next decoder after the current decoder is taken as the current decoder for the next convergence judgment. The step of obtaining the KV data and information matrix corresponding to the current decoder through the information matrix corresponding to the previous decoder before the current decoder in the large language model continues to be executed until the convergence condition is met. The data acquisition module is used to determine the data to be inferred based on the KV data corresponding to the current decoder that meets the convergence condition. The data to be inferred is used for model inference to determine the response text of the text to be processed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images