Knowledge injection-oriented large model reasoning method, electronic equipment and medium
By fusing word embeddings and hidden states in a large language model and applying windowing, combined with tree-based parallel verification, the reasoning delay caused by knowledge injection is solved, achieving efficient natural language processing tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-29
- Publication Date
- 2026-04-10
AI Technical Summary
In natural language processing tasks, existing technologies, especially in knowledge injection scenarios, suffer from inference latency and excessive computational load due to the introduction of long-tail entity information, rendering existing acceleration methods ineffective.
By fusing word embeddings and hidden states of a large language model, windowing and Transformer layer compression are employed, combined with a tree-based parallel verification strategy, to reduce redundant computation of lengthy contexts and to evaluate multiple candidate lexical paths in parallel.
It significantly improves response speed, with inference speed up to approximately 2.20 times faster, while maintaining prediction quality and reducing computational load, making it suitable for tasks such as object-level sentiment analysis and knowledge extraction.
Smart Images
Figure CN121835752A_ABST
Abstract
Description
Technical Field
[0001] This invention pertains to natural language processing, and particularly relates to a large-scale model reasoning method, electronic device, and medium oriented towards knowledge injection. Background Technology
[0002] With the widespread application of large-scale language models in natural language processing tasks, these models have demonstrated strong capabilities in areas such as text generation, question answering reasoning, and sentiment understanding. However, in project development and industrial deployment, large models themselves do not possess complete domain knowledge or long-tail entity information. They typically require the introduction of external knowledge, such as entity descriptions, dictionaries, domain hints, or long contextual references, to supplement their reasoning capabilities.
[0003] In most practical tasks (including but not limited to object-level sentiment analysis, knowledge extraction, and open-ended question answering), knowledge injection is usually achieved through extended input prompts, which explicitly incorporate background knowledge into the inference input sequence. The length of the prompts increases with the scale of knowledge, causing large models to process longer contexts during the inference phase. This significantly increases the inference latency of autoregressive decoding, severely impacting response speed and service throughput.
[0004] Existing inference acceleration methods include operator optimization, memory compression, model scaling, and speculative decoding. However, most of these methods only utilize word-level context or top-level hidden states when generating candidate words, which limits their predictive ability; some methods collect both types of information simultaneously but introduce additional computational overhead, making the acceleration effect insufficient to offset the cost.
[0005] Therefore, in knowledge injection scenarios, how to reduce inference time while maintaining prediction quality is a key problem that current technology urgently needs to solve. Summary of the Invention
[0006] To address the shortcomings of existing technologies, embodiments of the present invention provide a large-scale model reasoning method, electronic device, and medium oriented towards knowledge injection.
[0007] In a first aspect, embodiments of the present invention provide a large-model reasoning method oriented towards knowledge injection, the method comprising the following steps: Obtain a text sequence, and generate word embedding representations and hidden states for each word element in the text sequence using a large language model; The word embedding representation of each word element is fused with its corresponding hidden state to obtain the fused hidden representation; All blended hidden representations are divided according to a preset window size, and the blended hidden representations in each window are processed by the Transformer layer to obtain a compressed hidden state; All compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer are processed by the Transformer layer to obtain the candidate hidden state corresponding to each word. Based on the candidate hidden state corresponding to each word, the probability distribution of candidate words is output through the large language model, and multiple candidate words are obtained by sampling. Multiple candidate word sequences are constructed based on the aforementioned candidate words, and then input into the large language model through a tree-shaped attention mask to obtain the word output sequence.
[0008] In a second aspect, embodiments of the present invention provide an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-described knowledge-injection-oriented large model reasoning method.
[0009] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described knowledge-injection-oriented large model reasoning method.
[0010] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned knowledge-injection-oriented large model reasoning method.
[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: To address the issue of excessively long prompts in object-level sentiment analysis and information extraction, which require the injection of substantial external knowledge (such as entity descriptions and domain background), this invention introduces a context compression mechanism. By summarizing historical information, it avoids recalculating lengthy and complete contexts at each step of the prediction process. This significantly reduces the computational load while maintaining awareness of long-tail knowledge, thereby greatly improving response speed.
[0012] This invention employs a pluggable prediction module, eliminating the need for any modification or retraining of the weight parameters of the original target language model. As an independent plugin for the inference stage, this method can be directly integrated into various Transformer-based autoregressive models such as LLaMA and GPT, significantly reducing the technical deployment threshold and computational cost. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0014] Figure 1 A schematic diagram of a knowledge-injection-oriented large-model reasoning method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0016] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.
[0017] like Figure 1 As shown, this embodiment of the invention provides a large-scale model reasoning method oriented towards knowledge injection, the method comprising the following steps: Step S1: Obtain the text sequence, and generate the word embedding representation and hidden state corresponding to each word in the text sequence by the large language model.
[0018] Specifically, obtaining text sequences The text sequence includes task instructions, text to be processed, and external knowledge injection text, such as entity descriptions, domain descriptions, or dictionary expansions.
[0019] Step S2: Fuse the word embedding representation of each lexical unit with its corresponding hidden state to obtain the fused hidden representation.
[0020] Furthermore, the expression is as follows: ; In the formula, This represents the word embedding representation corresponding to the t-th lexical unit. This represents the hidden state corresponding to the t-th word. This represents the fused hidden representation corresponding to the t-th term. It is a linear mapping.
[0021] It should be noted that by fusing the word embedding representation of each lexical unit with its corresponding hidden state, a fused hidden representation is obtained, thereby reducing the subsequent computational burden while preserving word-level semantics and the model's internal inference state.
[0022] Step S3: Divide all the blended hidden representations according to the preset window size, and process the blended hidden representations in each window through the Transformer layer to obtain the compressed hidden state.
[0023] Specifically, set the window size to W; The fused hidden representation sequence is divided into consecutive windows according to window size W. Each window is processed by a Transformer layer to obtain the corresponding compressed hidden state; the expression is as follows: ; In the formula, This indicates the compressed / hidden state of the k-th window. This represents all merged hidden representations within the k-th window. This represents the Transformer decoding structure.
[0024] It should be noted that the present invention uses a windowing mechanism to compress historical hidden states in order to avoid repeatedly processing excessively long contexts. Employing a Transformer decoding structure, its attention mask ensures global visibility within windows while maintaining complete isolation between windows. It also compresses hidden states. The computation is cached after completion and can be reused across multiple inference steps, eliminating the need to repeatedly access all terms within the window during the inference phase. This significantly reduces computational costs, especially when knowledge injection results in extremely long texts.
[0025] Step S4: Process all compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer through the Transformer layer to obtain the candidate hidden state corresponding to each word.
[0026] Furthermore, after window compression, when the large language model generates the next lexical unit, it only combines the necessary contextual information for prediction. For the current last word... The next potential hidden state is calculated based on the compressed hidden states of all generated windows and the fused representation of the current window that has not yet been compressed, as shown in the following expression: ; In the formula, This represents the candidate hidden state corresponding to the (t+1)th lexical unit. The table represents the compressed / hidden state of the nth window. This represents all fused hidden representations that have not been processed by the Transformer layer. This represents the Transformer decoding structure.
[0027] It should be noted that, A Transformer decoding structure is used, but the masking mechanism strictly ensures that the current prediction depends only on information from past and current windows, without violating the autoregressive condition. The hidden state is compressed. As the key / value pair for the prediction layer Speculate(.), all fused hidden representations that have not been processed by the Transformer layer are computed as key / value / query to finally obtain the predicted candidate hidden states.
[0028] Step S5: Based on the candidate hidden state corresponding to each word, the probability distribution of candidate words is output through the large language model, and multiple candidate words are sampled.
[0029] Furthermore, the expression is as follows: ; In the formula, This represents the input text sequence. This represents the word sequence output by the large language model. This represents the candidate hidden state corresponding to the (t+1)th lexical unit. This represents the output header of the large language model.
[0030] Step S6: Construct multiple candidate word sequences based on the multiple candidate words, input them into the large language model through a tree-shaped attention mask, and obtain the word output sequence.
[0031] Specifically, this invention generates multiple candidate words and constructs multiple possible candidate lexical paths based on them. To avoid slowing down inference speed by verifying each candidate lexical path individually, this invention introduces a tree-structured parallel verification structure. Candidate lexical elements are expanded according to their Top-K depth to form a set of candidate lexical paths, which are then input into the target large language model via a tree-structured attention mask. The target large language model can evaluate multiple candidate lexical paths in parallel during a single inference operation. For consecutive positions... If the predicted probability of the target large language model satisfies the greedy acceptance criterion, then this invention directly uses the word sequence. As the final output, it is submitted all at once, thus reducing the number of recursive calls and transforming the original word-by-word iterative reasoning process into a step-by-step generation. The speed improvement brought by this process is particularly significant in knowledge injection scenarios, because the cost is the sequence processing overhead, while skipping multiple prediction positions at once can significantly offset the computational burden brought by long context.
[0032] It should be noted that this invention utilizes a tree attention mask to construct a multi-path verification strategy, enabling large models to simultaneously evaluate multiple candidate paths with only one forward propagation. Once multiple consecutive predicted words are accepted, the system can output multiple tokens in a single loop (step generation). Experiments show that this method can improve inference speed by up to approximately 2.20 times while ensuring that the output content is completely consistent with the original model, significantly shortening the generation time.
[0033] Furthermore, the present invention also includes training the prediction process, the training process including: Construct a joint loss function; the joint loss function is a weighted sum of the first loss function and the second loss function; the expression is as follows: In the formula, , All represent weighting coefficients; Wherein, the first loss function Using the Smooth L1 form, the predicted hidden state is required to approximate the hidden layer output of the target model. The expression is as follows: ; In the formula, This represents the hidden state corresponding to the (t+1)th word. This represents the candidate hidden state corresponding to the (t+1)th lexical unit; The second loss function To ensure that the predicted word probabilities match the actual word probabilities, the cross-entropy loss function is used, as shown in the following expression: ; ; In the formula, Represents the probability distribution of real lexical units. Represents the probability distribution of predicted terms. This represents the output header of the large language model. This represents the input text sequence. This represents the word sequence output by the large language model.
[0034] Furthermore, embodiments of the present invention also provide a knowledge-injection-oriented large-model reasoning system, the system being used to implement the above-described knowledge-injection-oriented large-model reasoning method, the system comprising: The feature extraction module is used to obtain the text sequence and the word embedding representation and hidden state corresponding to each word in the text sequence generated by the large language model; The fusion module is used to fuse the word embedding representation of each lexical unit with its corresponding hidden state to obtain a fused hidden representation; The context compression module is used to divide all the blended hidden representations according to a preset window size, and to process the blended hidden representations in each window through the Transformer layer to obtain the compressed hidden state; The prediction module is used to process all compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer to obtain the candidate hidden state corresponding to each word; based on the candidate hidden state corresponding to each word, the large language model outputs the probability distribution of candidate words and samples to obtain multiple candidate words. The verification module is used to construct multiple candidate word sequences based on the multiple candidate words, input them into the large language model through a tree-shaped attention mask, and obtain the word output sequence.
[0035] In summary, this invention, through fusion, compression, prediction, and parallel verification mechanisms, achieves a significant acceleration of the inference stage without modifying the original model parameters, while maintaining excellent response performance even under the contextual growth pressure caused by knowledge-injected text. This invention is applicable to natural language tasks with rich prompts, such as object-level sentiment analysis, knowledge extraction, multi-hop question answering, and any inference scenario that relies on external knowledge enhancement.
[0036] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the knowledge-injection-oriented large model reasoning method as described above. Figure 2 The diagram shown illustrates a hardware structure of any device with data processing capabilities for implementing the knowledge-injection-oriented large-model reasoning method provided in this embodiment of the invention, except... Figure 2 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0037] Accordingly, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the large model reasoning method for knowledge injection as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.
[0038] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.
[0039] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A large-scale model reasoning method oriented towards knowledge injection, characterized in that, The method includes: Obtain a text sequence, and generate word embedding representations and hidden states for each word element in the text sequence using a large language model; The word embedding representation of each word element is fused with its corresponding hidden state to obtain the fused hidden representation; All blended hidden representations are divided according to a preset window size, and the blended hidden representations in each window are processed by the Transformer layer to obtain a compressed hidden state; All compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer are processed by the Transformer layer to obtain the candidate hidden state corresponding to each word. Based on the candidate hidden state corresponding to each word, the probability distribution of candidate words is output through the large language model, and multiple candidate words are obtained by sampling. Multiple candidate word sequences are constructed based on the aforementioned candidate words, and then input into the large language model through a tree-shaped attention mask to obtain the word output sequence.
2. The knowledge-injection-oriented large-model reasoning method according to claim 1, characterized in that, The process of fusing the word embedding representation of each lexical unit with its corresponding hidden state to obtain the fused hidden representation includes: ； In the formula, This represents the word embedding representation corresponding to the t-th lexical unit. This represents the hidden state corresponding to the t-th word. This represents the fused hidden representation corresponding to the t-th term. It is a linear mapping.
3. The large-scale model reasoning method oriented towards knowledge injection according to claim 1, characterized in that, The process of dividing all blended hidden representations according to a preset window size, and then processing the blended hidden representations in each window through a Transformer layer to obtain a compressed hidden state includes: Set the window size to W; The fused hidden representation sequence is divided into consecutive windows according to window size W. Each window is processed by a Transformer layer to obtain the corresponding compressed hidden state; the expression is as follows: ； In the formula, This indicates the compressed / hidden state of the k-th window. This represents all merged hidden representations within the k-th window. This represents the Transformer decoding structure.
4. The knowledge-injection-oriented large-model reasoning method according to claim 1 or 3, characterized in that, The process of processing all compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer to obtain the candidate hidden state for each word includes: ； In the formula, This represents the candidate hidden state corresponding to the (t+1)th lexical unit. The table represents the compressed / hidden state of the nth window. This represents all fused hidden representations that have not been processed by the Transformer layer. This represents the Transformer decoding structure.
5. The knowledge-injection-oriented large-model reasoning method according to claim 1, characterized in that, The process of processing all compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer to obtain the candidate hidden state for each word includes: ； In the formula, This represents the input text sequence. This represents the word sequence output by the large language model. This represents the candidate hidden state corresponding to the (t+1)th lexical unit. This represents the output header of the large language model.
6. A large-scale model reasoning system oriented towards knowledge injection, characterized in that, The system is used to implement the knowledge-injection-oriented large model reasoning method as described in any one of claims 1-5, and the system includes: The feature extraction module is used to obtain the text sequence and the word embedding representation and hidden state corresponding to each word in the text sequence generated by the large language model; The fusion module is used to fuse the word embedding representation of each lexical unit with its corresponding hidden state to obtain a fused hidden representation; The context compression module is used to divide all blended hidden representations according to a preset window size, and to process the blended hidden representations in each window through the Transformer layer to obtain a compressed hidden state; The prediction module is used to process all compressed hidden states and all fused hidden representations that have not been processed by the Transformer layer to obtain the candidate hidden state corresponding to each word; based on the candidate hidden state corresponding to each word, the large language model outputs the probability distribution of candidate words and samples to obtain multiple candidate words. The verification module is used to construct multiple candidate word sequences based on the multiple candidate words, input them into the large language model through a tree-shaped attention mask, and obtain the word output sequence.
7. The knowledge-injection-oriented large-model reasoning system according to claim 6, characterized in that, The training process of the prediction module includes: Construct a joint loss function; the joint loss function is a weighted sum of the first loss function and the second loss function; Wherein, the first loss function The expression is as follows: ； In the formula, This represents the hidden state corresponding to the (t+1)th word. This represents the candidate hidden state corresponding to the (t+1)th lexical unit; The second loss function The expression is as follows: ； ； In the formula, Represents the probability distribution of real lexical units. Represents the probability distribution of predicted terms. This represents the output header of the large language model. This represents the input text sequence. This represents the word sequence output by the large language model.
8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the knowledge-injection-oriented large model reasoning method as described in any one of claims 1-5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the knowledge-injection-oriented large model reasoning method as described in any one of claims 1-5.
10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the knowledge-injection-oriented large model reasoning method as described in any one of claims 1-5.