Big language model reasoning method and device

Abstract

The invention provides a big language model reasoning method and device, and relates to the technical field of artificial intelligence and natural language processing. The reasoning method comprises the following steps: mapping a problem sequence into a problem embedding matrix through an embedding layer, and mapping a context sequence into a context embedding matrix; compressing the context embedding matrix through a compression layer to obtain a compressed context embedding matrix; the parameter quantity of the compressed context embedding matrix is smaller than that of the context embedding matrix; generating an input embedding matrix according to the compressed context embedding matrix and the problem embedding matrix; and inputting the input embedded matrix into a Transform layer to obtain a reasoning result of the Transform layer. According to the method, the original context embedding matrix is compressed through the compression layer, and the compressed context embedding matrix and the problem embedding matrix are used for subsequent reasoning calculation, so that the number of tokens participating in the reasoning calculation subsequently is reduced, and the reasoning efficiency is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and more specifically, to a reasoning method and apparatus for a large language model. Background Technology

[0002] As large language models are increasingly used in tasks such as enhanced question answering, long text summarization, and long sequence reasoning, the length of context that these models need to process is also constantly increasing. Currently, in the reasoning process based on the Transformer architecture, the computational complexity of the attention mechanism typically increases quadratically or nearly quadratically with the context length N, which has become a major bottleneck restricting the efficient reasoning of the model. With the increase of N, subsequent reasoning computations become large and reasoning efficiency is low. Summary of the Invention

[0003] The purpose of this application is to provide a reasoning method and apparatus for large language models, so as to reduce the amount of reasoning computation and improve reasoning efficiency.

[0004] In a first aspect, embodiments of this application provide a reasoning method for a large language model, the large language model including an embedding layer, a compression layer, and a Transformer layer; the reasoning method includes: The question sequence is mapped to a question embedding matrix and the context sequence is mapped to a context embedding matrix through the embedding layer; The context embedding matrix is ​​compressed using a compression layer to obtain a compressed context embedding matrix; the compressed context embedding matrix has fewer parameters than the original context embedding matrix. Generate the input embedding matrix based on the compressed context embedding matrix and the question embedding matrix; Input the embedding matrix into the Transformer layer to obtain the inference result of the Transformer layer.

[0005] This application embodiment compresses the original context embedding matrix through a compression layer, and then uses the compressed context embedding matrix and the question embedding matrix for subsequent reasoning calculations, which greatly reduces the number of tokens involved in subsequent reasoning calculations and thus improves reasoning efficiency.

[0006] In one possible implementation of the first aspect, the context embedding matrix is ​​compressed using a compression layer to obtain a compressed context embedding matrix, including: Obtain the compressed query parameter matrix; the compressed query parameter matrix is ​​obtained after training the compressed layer; Project the compressed query parameter matrix and the context embedding matrix into the same attention space; Multi-head attention is calculated based on the projected compressed query parameter matrix and context embedding matrix to obtain a compressed output representation; Self-attention calculation is performed on the compressed output representation to obtain the compressed context embedding matrix.

[0007] In this embodiment, the compressed query parameter matrix and the context embedding matrix are subjected to multi-head attention calculation through a compression layer to obtain a compressed output representation, thereby realizing the aggregation of context information.

[0008] In one possible implementation of the first aspect, after obtaining the compressed query parameter matrix, the inference method further includes: Obtain the conditional bias; the conditional bias is obtained by pooling and linear mapping the problem embedding matrix. The conditionalized compressed query parameter matrix is ​​obtained based on the conditional bias and the compressed query parameter matrix. Accordingly, the compressed query parameter matrix and the context embedding matrix are projected into the same attention space, including: The conditional compressed query parameter matrix and context embedding matrix are projected into the same attention space.

[0009] This application embodiment obtains conditional bias by performing a linear mapping on the question embedding matrix and introduces conditional bias into the compressed query parameter matrix, so that the compression result is related to the current question, thereby improving reasoning efficiency while ensuring a certain degree of reasoning accuracy.

[0010] In one possible implementation of the first aspect, after obtaining the compressed query parameter matrix, the inference method further includes: Obtain the compression position code corresponding to the compression query parameter matrix; The conditionalized compressed query parameter matrix is ​​obtained based on the conditional bias, compressed position encoding, and compressed query parameter matrix.

[0011] In this embodiment, by introducing compressed position encoding into the compressed query parameter matrix, the order of the obtained compressed context embedding matrix is ​​consistent with or approximately consistent with the original order.

[0012] In one possible implementation of the first aspect, generating the input embedding matrix based on the compressed context embedding matrix and the question embedding matrix includes: A preset number of tokens are selected from the token sequence corresponding to the context embedding matrix according to preset selection rules; The input embedding matrix is ​​obtained by concatenating the original embedding matrix, the compressed context embedding matrix, and the question embedding matrix corresponding to a preset number of tokens.

[0013] In this embodiment, the compressed context embedding matrix and the original embedding matrix corresponding to a preset number of tokens are used as the input embedding matrix. The compressed context embedding matrix provides global convergence capability, and the preset number of tokens provides local precise information and reference anchors. The combination of the two can maintain inference performance more stably under high compression ratio.

[0014] In one possible implementation of the first aspect, the preset selection rules include: Obtain the relevance score for each token in the token sequence, and select a preset number of tokens starting from the highest relevance score; or, Select a preset number of tokens based on the position information of each token in the token sequence; or, By combining relevance scores and location information, a preset number of tokens are selected from the token sequence.

[0015] In this application embodiment, a preset number of tokens are selected by a preset selection rule. These preset number of tokens are considered to contain important contextual information. The original embedding matrix corresponding to the selected preset number of tokens is used as part of the input embedding matrix, which can provide local accurate information and reference anchors, thereby improving the accuracy of inference.

[0016] In one possible implementation of the first aspect, the original embedding matrix, the compressed context embedding matrix, and the question embedding matrix corresponding to a predetermined number of tokens are concatenated to obtain the input embedding matrix, including: If compression is enabled, the original embedding matrix, the compressed context embedding matrix, and the question embedding matrix corresponding to a preset number of tokens will be concatenated to obtain the input embedding matrix. If compression is not enabled, the original embedding matrix and the question embedding matrix corresponding to the preset number of tokens will be concatenated to obtain the input embedding matrix.

[0017] The compression layer in this embodiment can be considered pluggable, that is, whether to enable compression of the context embedding matrix can be selected as needed. When enabled, the original embedding matrix corresponding to a preset number of tokens, the compressed context embedding matrix and the question embedding matrix are concatenated to obtain the input embedding matrix, thereby meeting different user needs.

[0018] In one possible implementation of the first aspect, the compressed context embedding matrix includes position estimation information, which is obtained by weighted summation of the position codes corresponding to the context embedding matrix based on the cross-attention weights of the compression layer; before generating the input embedding matrix, the method further includes: The vectors in the compressed context embedding matrix are reordered based on the location estimation information.

[0019] This application embodiment reorders the vectors in the compressed context embedding matrix based on position estimation information, so that the order of the compressed context embedding matrix input to the Transformer layer is consistent with the original token order, thereby improving the interpretability and position consistency of the compressed sequence.

[0020] In one possible implementation of the first aspect, the method further includes: Obtain training samples, which include a training context embedding matrix, a training question matrix, and a training answer matrix; The training context embedding matrix and the training question matrix input parameters are frozen to train a large language model, and the first prediction result of the training large language model is obtained. Input the training context embedding matrix into the compressed model to be trained, and obtain the compressed training context embedding matrix output by the compressed model to be trained. The training large language model is trained by freezing the input parameters of the compressed context embedding matrix and the training question matrix, and then the second prediction result of the training large language model is obtained. Calculate the KL divergence loss based on the first and second prediction results; Calculate the cross-entropy loss based on the second prediction result and the training answer matrix; The total loss is determined based on the KL divergence loss and cross-entropy loss. The parameters of the compressed model to be trained are optimized based on the total loss; where the compressed model after training is the compression layer.

[0021] In this embodiment, the parameters of the compressed model are optimized by using KL divergence loss and cross-entropy loss. KL divergence loss allows the output probability distribution of the student model to be as close as possible to the output probability distribution of the teacher model, while cross-entropy loss allows the student's prediction to be as close as possible to the true label of the data, thereby improving the performance of the compressed model.

[0022] Secondly, embodiments of this application provide an inference apparatus for a large language model, the large language model including an embedding layer, a compression layer, and a Transformer layer; the apparatus includes a processing module, which is used to: map a question sequence to a question embedding matrix through the embedding layer; map a context sequence to a context embedding matrix; compress the context embedding matrix through the compression layer to obtain a compressed context embedding matrix; wherein the number of parameters of the compressed context embedding matrix is ​​less than the number of parameters of the context embedding matrix; generate an input embedding matrix based on the compressed context embedding matrix and the question embedding matrix; and input the input embedding matrix into the Transformer layer to obtain the inference result of the Transformer layer.

[0023] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein: The processor and memory communicate with each other via a bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0024] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising: A non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the methods in the various possible implementations of the first aspect.

[0025] Fifthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the methods in various possible implementations of the first aspect.

[0026] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram illustrating a reasoning method for a large language model provided in an embodiment of this application; Figure 2This is a schematic diagram of another large language model inference method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a model training method provided in an embodiment of this application; Figure 4 Flowchart of another method for obtaining an input embedding matrix provided in an embodiment of this application; Figure 5 This is a schematic flowchart of a compression layer processing method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0031] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0033] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0034] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0035] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0036] As large language models are increasingly used in tasks such as enhanced question answering, long text summarization, and long sequence reasoning, the length of the context that these models need to process continues to grow. The attention computation of existing Transformer-based inference models typically increases quadratically or nearly quadratically with the context length N. This growth in context leads to: 1) extremely high computational cost in the prefill stage (one-time forward pass of long context); 2) while the decode stage (autoregressive generation) has smaller increments per step, it requires continuous access to the long context key-value cache; and 3) directly truncating the context significantly impairs accuracy, especially in multi-hop reasoning and fact recall tasks. Therefore, a technical solution is needed that can compress long contexts into shorter representations to improve inference speed while maintaining performance as much as possible.

[0037] This application proposes a reasoning method for a large language model. This method adds a compression layer to the traditional large language model, positioned between the embedding layer and the Transformer layer. The compression layer compresses the context embedding matrix to obtain an embedding matrix with fewer parameters. Compression refers to feeding the context embedding matrix into a neural network, where the network outputs a predetermined number of vectors to form a new matrix—the compressed context embedding matrix. After obtaining the compressed context embedding matrix, it is concatenated with the question embedding matrix and then input into the Transformer layer to obtain the inference result. Compared to the uncompressed embedding matrix, the embedding matrix input to the Transformer layer has significantly fewer parameters, thus reducing subsequent computation and improving inference efficiency.

[0038] It is understood that the reasoning method provided in this application embodiment can be applied to electronic devices, which include terminals and servers; wherein the terminal may specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; and the server may specifically be an application server or a web server.

[0039] Figure 1 This is a schematic flowchart illustrating a reasoning method for a large language model provided in an embodiment of this application. The method includes: Step 101: Map the question sequence to a question embedding matrix and the context sequence to a context embedding matrix using the embedding layer. It can be understood that the mapping of the question sequence and the mapping of the context sequence can be performed in parallel, or the mapping of the question sequence can be performed first, followed by the mapping of the context sequence, or vice versa.

[0040] in, Figure 2 This is a schematic diagram of another large language model reasoning method provided in an embodiment of this application. The context sequence and question sequence are obtained in the following way: The text to be inferred is obtained and divided into context text and question text. Context text is the background information text upon which the large language model relies for inference, such as an article, a dialogue, or a scene description. The context sequence can come from a dataset, user input, or real-time acquisition. Question text is the specific query text proposed based on the context. It can be a text question entered by the user or language entered by the user, and then the language is converted into a text question.

[0041] The obtained context text and question text undergo preprocessing to improve tokenization quality. Preprocessing includes removing redundant spaces, line breaks, special characters, or irrelevant punctuation, as well as unifying capitalization. Furthermore, if the context text and question text are too long and exceed the input limits of the large language model, they can be truncated or segmented to accommodate the model's input text length requirements.

[0042] After preprocessing, a tokenizer can be used to segment the preprocessed text into tokens. Tokens are the basic units of text and can be words, subwords, or characters, depending on the chosen model or tokenization method. For example, for English text, tokenizers such as BERT's WordPiece or GPT's Byte Pair Encoding can be used; for Chinese text, character-based or word-based word segmentation tools can be employed.

[0043] The tokens after word segmentation are mapped to positive indices in the vocabulary to form a numerical sequence. Specifically, the context text is mapped to generate a context sequence, and the question text is mapped to generate a question sequence. Each token corresponds to a unique ID, facilitating mathematical operations within the large language model.

[0044] The embedding layer is essentially a learnable lookup table, the size of which is [vocabulary size, embedding dimension], where the embedding dimension is a preset hyperparameter, such as 768 or 1024. During model training, this embedding layer has been trained with a large amount of text, and the row vector corresponding to each token ID has encoded rich semantic and syntactic information.

[0045] By performing a lookup operation, the corresponding d-dimensional row vector is retrieved from the embedding table for each token ID in the sequence. Therefore, for the question sequence, the corresponding question embedding matrix can be obtained, and for the context sequence, the context embedding matrix can be obtained. Here, d represents the hidden dimension of the large language model.

[0046] Step 102: Compress the context embedding matrix through a compression layer to obtain the compressed context embedding matrix.

[0047] The compression layer can be a small neural network model, or its architecture can be a Transformer. This compression layer can compress the input context embedding matrix and output a compressed context matrix. The goal is to reduce the length of the input to the Transformer layer without losing key information, thereby reducing subsequent computational complexity and improving efficiency.

[0048] Compression layers can aggregate local features through average pooling or max pooling to compress the context embedding matrix and reduce sequence length; or attention pooling can be used to adaptively weight and select important information through a learnable attention mechanism; in addition, linear projection or convolution can also be used to compress the context embedding matrix.

[0049] The compressed context embedding matrix has fewer parameters than the original context embedding matrix. For example, before compression, there is a context embedding matrix for 1024 tokens. After compression, 32 tokens can be selected from these 1024 tokens, and their corresponding embedding vectors can be used to construct the compressed context embedding matrix. For ease of understanding, the compression layer can be considered to contain M slots. The purpose of this compression layer is to select M tokens from the 1024 tokens and load their vector representations into these M slots, with each slot containing one vector representation, resulting in M ​​slot representations.

[0050] It should be noted that the compression layer can be pre-trained before using a large language model for reasoning. Figure 3 This application provides a schematic diagram of a model training method, the training process of which is as follows: To obtain training samples, for example, a recursive scan of the training data directory is performed to obtain files with various suffixes, such as json, jsonl, gz, etc.

[0051] After obtaining the training samples, the file list is split into shards according to the distributed rank and dataloaderworker process; random shuffling and buffered sampling are performed within each shard to improve data diversity and reduce memory usage.

[0052] The training samples include a training context embedding matrix, a training question matrix, and a training answer matrix. The training context embedding matrix is ​​a dense vector representation of the original training context text sequence after an embedding layer transformation, with a shape of [sequence length L_c, embedding dimension d]. It contains rich semantic and positional information. The corresponding question text sequence is a vector representation after the same embedding layer transformation, with a shape of [sequence length L_q, embedding dimension d]. The label representation of the true answers (i.e., the training answer matrix) is typically a one-hot encoded vector corresponding to the vocabulary or options.

[0053] The training context embedding matrix and training question embedding matrix are input into a large language model whose parameters are frozen, and the first prediction result is obtained from the output of the large language model. For example, the uncompressed training context embedding matrix and training question embedding matrix are input together into the large language model, which acts as a teacher model, and its parameters are fixed (frozen) and not updated. The teacher model processes and infers based on complete contextual information and outputs a probability distribution, i.e., the first prediction result. This result is considered a more accurate and reliable soft label because it is based on all information.

[0054] The training context embedding matrix is ​​input into the compressed model to be trained, resulting in the compressed training context embedding matrix output by the compressed model. For example, the training context embedding matrix is ​​input into the compressed model to be trained. This compressed model is typically a lightweight neural network (such as a small transformer, convolutional network, or dedicated adapter). The compressed model processes the context embedding matrix with the goal of learning an efficient information summarization representation, outputting a compressed training context embedding matrix, whose shape is typically [compression length M, embedding dimension d], where M < d. <L_c。

[0055] The compressed context embedding matrix and the training question matrix are input into a large training language model with frozen parameters to obtain the second prediction result output by the large training language model. For example, the compressed context embedding matrix and the training question embedding matrix are concatenated and input into the same large training language model. In this case, the large training language model acts as a student model, and its parameters remain fixed. The student model can only reason based on the limited compressed context information and output another probability distribution, i.e., the second prediction result.

[0056] Calculate the KL divergence loss based on the first and second prediction results, and calculate the cross-entropy loss based on the second prediction result and the training answer matrix.

[0057] For example, the difference between the first prediction (teacher output) and the second prediction (student output) is compared using KL divergence as a metric. KL divergence measures the information loss between one probability distribution and another reference distribution. The optimization objective here is to minimize KL divergence, which forces the contextual representation of the compressed model's output to make the prediction distribution produced by the large language model as close as possible to its prediction distribution produced in the full context. This is equivalent to distilling the teacher model's knowledge (soft judgments about the question) into the compressed model. Simultaneously, it is also necessary to ensure that the prediction results using the compressed context are accurate. Therefore, the second prediction result is compared with the true training answer matrix, and standard cross-entropy loss is calculated. This loss ensures that the final output of the entire system still optimizes towards the correct answer, even under information compression.

[0058] The total loss is determined based on the KL divergence loss and cross-entropy loss. The total loss is a weighted sum of the knowledge distillation loss and the task loss, i.e.: Total Loss = α KL divergence loss + β Cross-entropy loss. α and β are hyperparameters used to balance the goals of mimicking the teacher and answering questions correctly.

[0059] The parameters of the compressed model to be trained are optimized based on the total loss; the trained compressed model is the compression layer. During backpropagation, gradients are calculated and optimized based on this total loss. The parameters of the large language model (teacher / student) remain constant throughout the entire process.

[0060] After repeated iterative training through the above process, the compressed model to be trained learns how to extract the most essential and critical information for the question-answering task from the original context and encode it into a compact representation.

[0061] After training, this optimized model serves as the final compression layer, which can be deployed during the inference phase to efficiently compress any new context, thereby significantly reducing computational and memory overhead while maintaining the overall performance of large language models. It should be noted that this compression layer is pluggable, meaning that enabling compression of the context embedding matrix can be selected. Of course, this compression layer can also be used specifically for other large language models.

[0062] It should be noted that, for cases with a limited number of training samples, a greedy distillation algorithm can be used for training. The teacher generates a sequence of logits within a complete context, and this sequence is used as the distillation target to optimize prediction consistency under student conditions.

[0063] Step 103: Generate the input embedding matrix based on the compressed context embedding matrix and the question embedding matrix.

[0064] See Figure 2 The compressed context embedding matrix is ​​concatenated with the question embedding matrix to obtain the input embedding matrix. For example, the question embedding matrix can be concatenated after the compressed context embedding matrix, or it can be concatenated before the compressed context embedding matrix.

[0065] Step 104: Input the embedding matrix into the Transformer layer to obtain the inference result of the Transformer layer. See also Figure 2 The electronic device can also record the inference performance of this operation, including, for example, prefill latency, decode latency, and compression ratio. The recorded inference performance can be used for subsequent optimization of the compression layer.

[0066] This application embodiment compresses the original context embedding matrix through a compression layer, and then uses the compressed context embedding matrix and the question embedding matrix for subsequent reasoning calculations, which greatly reduces the number of tokens involved in subsequent reasoning calculations and thus improves reasoning efficiency.

[0067] Based on the above embodiments, the detailed process of the compression layer compressing the context embedding matrix is ​​as follows: Obtain the compressed query parameter matrix; where the compressed query parameter matrix is ​​a set of learnable query vectors used to extract and aggregate information from the context, also known as slots. The compressed query parameter matrix can be obtained by training the compression layer. Each row represents the initial query vector for a specific slot, which can be understood as a series of questions or empty templates used to ask questions to the context in order to extract specific types of information, such as: what is the location, who is the person.

[0068] The compressed query parameter matrix and the context embedding matrix are projected into the same attention space. For example, representations from different sources (including the learnable compressed query parameter matrix and the context embedding matrix) are projected into the same feature dimension and semantic space through independent linear transformations. This is to enable subsequent attention mechanisms to perform efficient similarity calculations and information exchange.

[0069] Multi-head attention computation is performed based on the projected compressed query parameter matrix and the context embedding matrix to obtain a compressed output representation. The compressed query parameter matrix includes multiple compressed query vectors. The projected compressed query vectors review the entire context embedding matrix and calculate which parts of the context information are most relevant to them through attention weights. Then, these relevant parts are weighted and aggregated. The multi-head mechanism allows the model to pay attention to the context in parallel from different representation subspaces, thereby capturing richer information.

[0070] Self-attention computation is performed on the compressed output representation to obtain the compressed context embedding matrix. The purpose of this step is to enable communication and comparison between the various compressed representations, thereby resolving dependencies or conflicts between the compressed output representations and integrating global information to make the obtained compressed context embedding matrix more accurate and consistent.

[0071] In this embodiment, the compressed query parameter matrix and the context embedding matrix are subjected to multi-head attention calculation through a compression layer to obtain a compressed output representation, thereby realizing the aggregation of context information.

[0072] Building upon the above embodiments, to ensure the obtained compressed context embedding matrix is ​​relevant to the current problem, the problem embedding matrix can be conditionalized to obtain a conditional bias. This conditional bias is then incorporated into the compressed query parameter matrix to obtain a conditionalized compressed query parameter matrix. Subsequently, the conditionalized compressed query parameter matrix is ​​used for attention calculation.

[0073] The conditional bias is obtained through the following method: Pooling is performed on the question embedding matrix to obtain question vectors. These question vectors are then linearly mapped to obtain conditional biases. The question embedding matrix can be represented as [L_q, d], where L_q is the length of the question sequence and d is the embedding dimension. This matrix contains the semantic and positional information of each word in the question. Pooling is performed along the sequence length dimension (L_q). For example, average pooling can be used, which calculates the average of all word vectors. Max pooling or summation pooling can also be used.

[0074] A general problem semantic summary is projected onto a form and space required for a specific task or module through a learnable linear layer (fully connected layer). The result of this mapping is the conditional bias. It can be viewed as a bias term or modulation signal dynamically generated from the current problem content. The weights and bias parameters of this linear layer are learned during training.

[0075] After obtaining the conditionalized compressed query parameter matrix, the conditionalized compressed query parameter matrix and the context embedding matrix are projected into the same attention space, and then multi-head attention cross-computation is performed based on the conditionalized compressed query parameter matrix and the context embedding matrix projected into the same attention space.

[0076] This application embodiment obtains conditional bias by linearly mapping the question embedding matrix and introducing conditional bias into the compressed query parameter matrix. This is equivalent to injecting the semantic information of the current question into the originally general slot query, so that the subsequent cross-attention aggregation can adaptively focus on the context content related to the question. Thus, during the compression process, information that is crucial to answering the current question is retained first, thereby improving reasoning efficiency while ensuring a certain degree of reasoning accuracy.

[0077] Building upon the above embodiments, to maintain the original order of the compressed context embedding matrix, a compressed positional encoding can be assigned to the compressed query parameter matrix. This compressed query parameter matrix can be denoted as Q_comp. Its shape is [M, d], where M is the preset length (i.e., slots) of the compressed representation, and M is the feature dimension. Each row of the matrix represents a basic query vector used to extract and summarize information from the original context. The compressed positional encoding is a learnable positional encoding matrix, denoted as P_comp, with the same shape as Q_comp, [M, d]. The compressed positional encoding assigns explicit positional or order information to the M basic query vectors in Q_comp. Although these queries are logically parallel, the compressed representation (consisting of the results of these M queries) may inherently possess an order or structural relationship. The learnable P_comp allows the model to autonomously grasp this relative positional relationship within the compressed representation space.

[0078] Adding the conditional bias, compressed positional encoding, and compressed query parameter matrix yields the conditionalized compressed query parameter matrix. The compressed query parameter matrix provides a basic template for information extraction, defining the approximate type of information to be sought; for example, the first query might focus on capturing the main idea, while the second focuses on details. However, it is not yet bound to a specific question or sequence. The conditional bias is a dynamic bias obtained by mapping the current question vector. The compressed positional encoding injects each basic query with its specific positional identity within the compressed representation sequence. This ensures that the final M compressed representation vectors have diverse roles and an ordered structure.

[0079] In this embodiment, compressed position encoding is introduced into the compressed query parameter matrix, so that the order of the obtained compressed context embedding matrix is ​​consistent with the original order.

[0080] Based on the above embodiments, in addition to the compressed context embedding matrix and the question embedding matrix, the input embedding matrix may also include embedding matrices corresponding to a preset number of tokens selected from the original token sequence, as follows: According to preset selection rules, a preset number of tokens are selected from the token sequence corresponding to the context embedding matrix (i.e., the original token sequence). The original embedding matrix, the compressed context embedding matrix, and the question embedding matrix corresponding to the preset number of tokens are concatenated to obtain the input embedding matrix. The concatenation can be done sequentially in the order of the original embedding matrix, the compressed context embedding matrix, and the question embedding matrix, but other orders are also possible. This embodiment does not specify a particular concatenation order. The preset number is less than the number of tokens contained in the original token sequence. It is understood that a larger preset number results in more useful information input to the Transformer layer, higher inference accuracy, and correspondingly, a greater computational load. A smaller preset number results in less data input to the Transformer layer, less computation, faster inference speed, and correspondingly, lower inference accuracy. Therefore, a reasonable value can be set according to actual needs.

[0081] Assuming the preset quantity is n, the preset selection rules include any of the following: (1) Obtain the relevance score for each token in the token sequence, and select n tokens starting from the maximum relevance score. The relevance score can be obtained by performing a dot product or cosine similarity calculation between the pooling vector of the question embedding matrix (i.e., the question vector used to generate the conditional bias) and the embedding of each context token. Alternatively, during training, the attention weight distribution generated by the large language model (teacher model) on the full context (e.g., the attention of the last layer [CLS] tokens to the context) can be used as a proxy for the relevance score.

[0082] (2) Select a preset number of tokens based on the position information of each token in the token sequence; for example, you can start from the beginning of the token sequence and select a preset number of tokens; or you can select n1 tokens from the beginning, n2 tokens from the end, and n3 tokens from certain specific positions in the middle, n1+n2+n3=n. This is based on the prior knowledge that the beginning and end of an article often contain the main idea and conclusion.

[0083] (3) Combining the relevance score and location information, select a preset number of tokens from the token sequence. For example, a candidate pool can be set up, such as first selecting 2n candidate tokens based on the location rule. Among these 2n candidate tokens, they are then sorted according to their relevance score to the question, and the top n are selected.

[0084] After the selection is completed, a list containing n original token IDs is obtained. Based on this list, n d-dimensional vectors are extracted from the original context embedding matrix to form the original key token embedding matrix (i.e., the original embedding matrix), which has the shape [n, d].

[0085] In this embodiment, the compressed context embedding matrix and the original embedding matrix corresponding to a preset number of tokens are used as the input embedding matrix. The compressed context embedding matrix provides global convergence capability, and the preset number of tokens provides local precise information and reference anchors. The combination of the two can maintain inference performance more stably under high compression ratio.

[0086] Figure 4 This is a flowchart illustrating another method for obtaining the input embedding matrix provided in an embodiment of this application. The original embedding matrix shown in this diagram is selected based on relevance; therefore, in this diagram, the original embedding matrix is ​​obtained after the cross-attention calculation. It can be understood that, based on prior rules, the original embedding matrix can be obtained before the cross-attention calculation. In this embodiment, the method for obtaining the input embedding matrix is ​​as follows: After cross-attention calculation, the context embedding matrix is ​​compressed to obtain the compressed context embedding matrix. It should be noted that the specific calculation method for the compressed context embedding matrix can be found in the above embodiment. Based on the cross-attention calculation result, K tokens are selected from the original token sequence, and the embedding vectors corresponding to the K tokens are concatenated to obtain the original embedding matrix.

[0087] After obtaining the compressed context embedding matrix, it is rearranged based on the position information. After rearrangement, it is determined whether the compression function is enabled. If enabled, the original embedding matrix corresponding to a preset number of tokens, the compressed context embedding matrix, and the question embedding matrix are concatenated to obtain the input embedding matrix. If the compression function is not enabled, the original embedding matrix corresponding to a preset number of tokens and the question embedding matrix are concatenated to obtain the input embedding matrix.

[0088] The compression layer in this embodiment can be considered pluggable, that is, whether to enable compression of the context embedding matrix can be selected as needed. When enabled, the original embedding matrix corresponding to a preset number of tokens, the compressed context embedding matrix and the question embedding matrix are concatenated to obtain the input embedding matrix, thereby meeting different user needs.

[0089] Figure 5 This is a schematic flowchart of a compression layer processing method provided in an embodiment of this application. The specific method is as follows: (1) Obtain the compressed query parameter matrix Q and introduce the compressed position code into the compressed query parameter matrix Q, that is, add the compressed query parameter matrix Q and the compressed position code.

[0090] (2) Obtain the question embedding matrix, perform pooling on the question embedding matrix to obtain the question vector. Then perform linear mapping on the question vector to obtain the conditional bias q_bias.

[0091] (3) The compressed query parameter matrix Q, which incorporates compressed position encoding, is added to the conditional bias q_bias to obtain the conditional compressed query parameter matrix q_proj.

[0092] (4) Obtain the key k_proj and value v_proj based on the context embedding matrix. The key k_proj introduces a positional encoding, which can be understood as being applied after the multi-head cross-attention calculation.

[0093] (5) Perform multi-head cross-attention calculation based on q_proj, k_proj and v_proj to obtain attention scores.

[0094] (6) The attention scores are linearly normalized, self-attention is calculated and linearly normalized respectively to realize the information interaction and re-expression between the compressed context embedding vectors, thereby improving the quality of the compressed context embedding matrix.

[0095] (7) Stabilize the numerical scale of training and inference through a learnable scaling factor and provide a switch compatible with old checkpoints. For example, a learnable scaling parameter is introduced at key computational nodes of the neural network model (e.g., after the normalization layer). During model training, this scaling parameter is automatically updated through a backpropagation algorithm (e.g., gradient descent). Its function is to constrain the numerical range of intermediate activation values ​​within a stable interval, thereby effectively suppressing gradient explosion or gradient vanishing phenomena and improving the convergence of model training and the consistency of inference results.

[0096] (8) Obtain the compressed context embedding matrix, which includes position estimation information. The position estimation information is obtained by weighted summation of the position codes corresponding to the context embedding matrix based on the cross attention weights of the compression layer. Sort each context embedding vector in the context embedding matrix according to the position estimation information, so that the sorted context embedding vectors are consistent with or approximately consistent with the original context order, and obtain the input embedding matrix.

[0097] This application reorders the vectors in the compressed context embedding matrix based on position estimation information, so that the order of the compressed context embedding matrix input to the Transformer layer is consistent with the original token order. This is crucial for maintaining the temporal logic and causal order of the text, and can effectively avoid reasoning errors caused by compression disrupting the order, thereby improving the interpretability and positional consistency of the compressed sequence.

[0098] This application also provides an inference device for a large language model, which can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the various steps of the method embodiment can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The inference device includes a processing module, which is used to: map a question sequence to a question embedding matrix through an embedding layer; map a context sequence to a context embedding matrix; compress the context embedding matrix through a compression layer to obtain a compressed context embedding matrix; wherein the number of parameters in the compressed context embedding matrix is ​​less than the number of parameters in the context embedding matrix; generate an input embedding matrix based on the compressed context embedding matrix and the question embedding matrix; and input the input embedding matrix into a Transformer layer to obtain the inference result of the Transformer layer.

[0099] Based on the above embodiments, this processing module is specifically used to: obtain a compressed query parameter matrix; the compressed query parameter matrix is ​​obtained after training the compression layer; Project the compressed query parameter matrix and the context embedding matrix into the same attention space; Multi-head attention is calculated based on the projected compressed query parameter matrix and context embedding matrix to obtain a compressed output representation; Self-attention calculation is performed on the compressed output representation to obtain the compressed context embedding matrix.

[0100] Based on the above embodiments, the processing module is also used for: Obtain the conditional bias; the conditional bias is obtained by pooling and linear mapping the problem embedding matrix. The conditionalized compressed query parameter matrix is ​​obtained based on the conditional bias and the compressed query parameter matrix. The conditionalized compressed query parameter matrix and the context embedding matrix are projected into the same attention space.

[0101] Based on the above embodiments, the processing module is also used for: Obtain the compression position code corresponding to the compression query parameter matrix; The conditionalized compressed query parameter matrix is ​​obtained based on the conditional bias, the compressed position encoding, and the compressed query parameter matrix.

[0102] Based on the above embodiments, this processing module is specifically used for: A preset number of tokens are selected from the token sequence corresponding to the context embedding matrix according to a preset selection rule; The input embedding matrix is ​​obtained by concatenating the original embedding matrix corresponding to the preset number of tokens, the compressed context embedding matrix, and the question embedding matrix.

[0103] Based on the above embodiments, the preset selection rules include: Obtain the relevance score for each token in the token sequence, and select a preset number of tokens starting from the maximum relevance score; or, A preset number of tokens are selected based on the position information of each token in the token sequence; or, By combining the correlation score and the location information, a preset number of tokens are selected from the token sequence.

[0104] Based on the above embodiments, this processing module is specifically used for: If compression is enabled, the original embedding matrix corresponding to the preset number of tokens, the compressed context embedding matrix, and the question embedding matrix are concatenated to obtain the input embedding matrix. If the compression function is not enabled, the original embedding matrix corresponding to the preset number of tokens and the question embedding matrix are concatenated to obtain the input embedding matrix.

[0105] Based on the above embodiments, the compressed context embedding matrix includes position estimation information, which is obtained by weighted summation of the position codes corresponding to the context embedding matrix based on the cross-attention weights corresponding to the compression layer; the processing module is further used for: The vectors in the compressed context embedding matrix are reordered based on the location estimation information.

[0106] Based on the above embodiments, the inference device further includes a training module for: Obtain training samples, which include a training context embedding matrix, a training question matrix, and a training answer matrix; The training context embedding matrix and the training question matrix are frozen as input parameters to the training large language model, and the first prediction result is obtained from the output of the training large language model. The training context embedding matrix is ​​input into the compressed model to be trained to obtain the training compressed context embedding matrix output by the compressed model to be trained. The training large language model is used to freeze the training compressed context embedding matrix and the training question matrix as input parameters to obtain the second prediction result output by the training large language model. Calculate the KL divergence loss based on the first prediction result and the second prediction result; Calculate the cross-entropy loss based on the second prediction result and the training answer matrix; The total loss is determined based on the KL divergence loss and the cross-entropy loss; The parameters of the compressed model to be trained are optimized based on the total loss; wherein, the compressed model after training is the compressed layer.

[0107] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device includes: a processor 601, a memory 602, and a bus 603; wherein: The processor 601 and the memory 602 communicate with each other through the bus 603; The processor 601 is used to call program instructions in the memory 602 to execute the methods provided in the above-described method embodiments, such as: mapping a question sequence to a question embedding matrix through an embedding layer; mapping a context sequence to a context embedding matrix; compressing the context embedding matrix through a compression layer to obtain a compressed context embedding matrix; wherein the number of parameters in the compressed context embedding matrix is ​​less than the number of parameters in the context embedding matrix; generating an input embedding matrix based on the compressed context embedding matrix and the question embedding matrix; and inputting the input embedding matrix into a Transformer layer to obtain the inference result of the Transformer layer.

[0108] Processor 601 can be an integrated circuit chip with signal processing capabilities. The processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0109] The memory 602 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0110] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: mapping a question sequence to a question embedding matrix through an embedding layer; mapping a context sequence to a context embedding matrix; compressing the context embedding matrix through a compression layer to obtain a compressed context embedding matrix; wherein the number of parameters in the compressed context embedding matrix is ​​less than the number of parameters in the context embedding matrix; generating an input embedding matrix based on the compressed context embedding matrix and the question embedding matrix; and inputting the input embedding matrix into a Transformer layer to obtain the inference result of the Transformer layer.

[0111] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments. These instructions include, for example: mapping a question sequence to a question embedding matrix through an embedding layer; mapping a context sequence to a context embedding matrix; compressing the context embedding matrix through a compression layer to obtain a compressed context embedding matrix; wherein the number of parameters in the compressed context embedding matrix is ​​less than the number of parameters in the context embedding matrix; generating an input embedding matrix based on the compressed context embedding matrix and the question embedding matrix; and inputting the input embedding matrix into a Transformer layer to obtain the inference result of the Transformer layer.

[0112] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0113] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0115] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0116] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An inference method of a large language model, characterized by, The large language model includes an embedding layer, a compression layer, and a Transformer layer; the method includes: The embedding layer maps the question sequence to a question embedding matrix and the context sequence to a context embedding matrix. The context embedding matrix is ​​compressed using the compression layer to obtain a compressed context embedding matrix; wherein the number of parameters in the compressed context embedding matrix is ​​smaller than the number of parameters in the context embedding matrix. An input embedding matrix is ​​generated based on the compressed context embedding matrix and the question embedding matrix; The input embedding matrix is ​​input into the Transformer layer to obtain the inference result of the Transformer layer.

2. The inference method of claim 1, wherein, The step of compressing the context embedding matrix through the compression layer to obtain the compressed context embedding matrix includes: Obtain the compressed query parameter matrix; the compressed query parameter matrix is ​​obtained after training the compressed layer; Project the compressed query parameter matrix and the context embedding matrix into the same attention space; Multi-head attention is calculated based on the projected compressed query parameter matrix and context embedding matrix to obtain a compressed output representation; Self-attention calculation is performed on the compressed output representation to obtain the compressed context embedding matrix.

3. The inference method of claim 2, wherein, After obtaining the compressed query parameter matrix, the inference method further includes: Obtain the conditional bias; the conditional bias is obtained by pooling and linear mapping the problem embedding matrix. The conditionalized compressed query parameter matrix is ​​obtained based on the conditional bias and the compressed query parameter matrix. Accordingly, projecting the compressed query parameter matrix and the context embedding matrix into the same attention space includes: The conditionalized compressed query parameter matrix and the context embedding matrix are projected into the same attention space.

4. The inference method of claim 3, wherein, After obtaining the compressed query parameter matrix, the inference method further includes: Obtain the compression position code corresponding to the compression query parameter matrix; The conditionalized compressed query parameter matrix is ​​obtained based on the conditional bias, the compressed position encoding, and the compressed query parameter matrix.

5. The inference method of claim 1, wherein, The step of generating the input embedding matrix based on the compressed context embedding matrix and the question embedding matrix includes: A preset number of tokens are selected from the token sequence corresponding to the context embedding matrix according to a preset selection rule; The input embedding matrix is ​​obtained by concatenating the original embedding matrix corresponding to the preset number of tokens, the compressed context embedding matrix, and the question embedding matrix.

6. The inference method of claim 5, wherein, The preset selection rules include: Obtain the relevance score for each token in the token sequence, and select a preset number of tokens starting from the maximum relevance score; or, A preset number of tokens are selected based on the position information of each token in the token sequence; or, By combining the correlation score and the location information, a preset number of tokens are selected from the token sequence.

7. The inference method of claim 5, wherein, The step of concatenating the original embedding matrix corresponding to the preset number of tokens, the compressed context embedding matrix, and the question embedding matrix to obtain the input embedding matrix includes: If compression is enabled, the original embedding matrix corresponding to the preset number of tokens, the compressed context embedding matrix, and the question embedding matrix are concatenated to obtain the input embedding matrix. If the compression function is not enabled, the original embedding matrix corresponding to the preset number of tokens and the question embedding matrix are concatenated to obtain the input embedding matrix.

8. The inference method of claim 1, wherein, The compressed context embedding matrix includes position estimation information, which is obtained by weighted summation of the position encoding corresponding to the context embedding matrix based on the cross attention weights corresponding to the compression layer. Before generating the input embedding matrix, the method further includes: The vectors in the compressed context embedding matrix are reordered based on the location estimation information.

9. The inference method of any one of claims 1-8, wherein, The method further includes: Obtain training samples, which include a training context embedding matrix, a training question matrix, and a training answer matrix; The training context embedding matrix and the training question matrix are frozen as input parameters to the training large language model, and the first prediction result is obtained from the output of the training large language model. The training context embedding matrix is ​​input into the compressed model to be trained to obtain the training compressed context embedding matrix output by the compressed model to be trained. The training large language model is used to freeze the training compressed context embedding matrix and the training question matrix as input parameters to obtain the second prediction result output by the training large language model. Calculate the KL divergence loss based on the first prediction result and the second prediction result; Calculate the cross-entropy loss based on the second prediction result and the training answer matrix; The total loss is determined based on the KL divergence loss and the cross-entropy loss; The parameters of the compressed model to be trained are optimized based on the total loss; wherein, the compressed model after training is the compressed layer.

10. An inference apparatus of a large language model, comprising: The large language model includes an embedding layer, a compression layer, and a Transformer layer; the device includes a processing module, which is used to: map a question sequence into a question embedding matrix through the embedding layer; Map the context sequence to a context embedding matrix; The context embedding matrix is ​​compressed using the compression layer to obtain a compressed context embedding matrix; wherein the number of parameters in the compressed context embedding matrix is ​​smaller than the number of parameters in the context embedding matrix. An input embedding matrix is ​​generated based on the compressed context embedding matrix and the question embedding matrix; the input embedding matrix is ​​then input into the Transformer layer to obtain the inference result of the Transformer layer.

11. An electronic device, comprising: include: Processor, memory, and bus, among which: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-9 by calling the program instructions.

12. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-9.

13. A computer program product, characterised in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-9.

Images