Attention sink point-based large language model context capability excitation method and system

Abstract

The invention discloses a large language model context capability excitation method and system based on an attention sink point. The method comprises the following steps: firstly, extracting a key context information vector from input data; then, in some specific Transform decoding layers of the model, anchor lexical elements in the input sequence are identified, and attention calculation paths of the anchor lexical elements are separated. In the anchor point path, contextual information is injected into representation of anchor point lexical elements through one-time cross attention calculation, and the result is stored in a key value cache for being reused in the subsequent generation step; and the non-anchor lexical elements keep the original calculation mode. And finally, the model carries out autoregression lexical element generation based on the updated sequence representation until a complete text is output. As a universal module, the method can be seamlessly integrated to any pre-training large language model based on a Transform decoder, and context information of the model is effectively excited and utilized on the premise that extra reasoning calculation overhead is hardly introduced.

Description

Technical Field

[0001] This invention belongs to the field of large language model reasoning in artificial intelligence, and particularly relates to a method and system for stimulating the contextual capabilities of large language models based on attention focus. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have made significant progress in areas such as text understanding and multimodal interaction. However, large language models still face two major bottleneck problems when following complex or long instructions: "content illusion" and "context forgetting." The former refers to the model generating content that does not match the input facts, while the latter refers to the model forgetting the initial context information when processing long sequences of input.

[0003] Research indicates that one of the core root causes of the aforementioned problems lies in the inherent attention drift phenomenon in the Transformer architecture. During autoregressive generation, the model's attention weights naturally tend to favor newly generated tokens, while the attention to the initial tokens at the beginning of the sequence, which typically carry the most crucial contextual information (such as user commands, image features, etc.), gradually diminishes, leading to information loss. Researchers have also observed a common attention sink phenomenon in large language models: regardless of how the generation process progresses, the first token of the sequence (usually...) tends to sink. <bos>The word "beginning of sequence" consistently receives a significantly higher attention weight than other word terms.

[0004] Currently, solutions to the hallucination and contextual forgetting problems in large language models either rely on fine-tuning the model or require introducing complex external modules or cue engineering during the inference phase. However, these methods have certain limitations: while fine-tuning-based methods can alleviate hallucinations or enhance memory, they not only consume significant computational resources and training time but may also trigger "catastrophic forgetting"; intervention methods during the inference phase (such as complex cue engineering or repeatedly adjusting the model's internal state in each generation step) often introduce significant computational overhead, drastically reducing inference efficiency. Therefore, there is an urgent need for a technical solution that requires no training and has low computational overhead to address these challenges. Summary of the Invention

[0005] The technical problem to be solved by this invention is to stimulate the contextual capabilities of large language models without additional training and introducing significant computational overhead, thereby alleviating the problems of content illusion and context forgetting, and to provide a method and system for stimulating the contextual capabilities of large language models based on attention points.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution: In a first aspect, the present invention provides a method for stimulating the contextual capabilities of a large language model based on attention points, comprising the following steps: S1. Obtain the original input data according to the task requirements. After converting the original input data into an initial embedding sequence, extract the key context information that needs to be continuously monitored in the autoregressive lexical generation process from the initial embedding sequence to obtain the context information vector. S2. The initial embedding sequence is used as the input to the decoder of the pre-trained large language model. The decoder is composed of multiple Transformer decoding layers cascaded sequentially. Information injection is performed on a specific Transformer decoding layer. The specific processing flow in the Transformer decoding layer that performs information injection is as follows: First, the anchor word in the initial embedding sequence is identified, and the attention calculation paths of the anchor word and the other words are separated in the multi-head attention module of the Transformer decoding layer. In the attention calculation path of the anchor word, the context information vector is injected into the representation of the anchor word through an adaptive query mechanism to obtain a new representation of the anchor word. In the attention calculation path of the other words, the original attention calculation method of the large language model remains unchanged to obtain the representation of the other words. The new representation of the anchor word and the representation of the other words are re-concatenated according to the position of the word in the initial embedding sequence to form an updated sequence representation as the final result of the multi-head attention module of the Transformer decoding layer, thereby completing the processing of the Transformer decoding layer. S3. After processing by all Transformer decoding layers, the final sequence representation output by the decoder is used as the input to the language modeling head of the large language model to predict the probability distribution of the next word. A new word is sampled from this probability distribution and appended to the end of the text word ID sequence transformed from the input text to form the updated text word ID sequence. S4. Input the updated text lexical ID sequence into the large language model for autoregressive lexical generation until the complete output text is obtained.

[0007] Based on the above scheme, each step can be implemented in the following preferred manner.

[0008] As a preferred embodiment of the first aspect mentioned above, in S1, when the task type is a multimodal task, the original input data consists of images and text, and when the task type is a long dialogue task, the original input data contains only text.

[0009] As a preferred embodiment of the first aspect mentioned above, in S1, when the original input data contains images, the images are segmented and input into the visual encoder to obtain image embedding vectors; when the original input data contains text, the word segmenter of the large language model is used to convert the input text into a discrete text word ID sequence, and the text word ID sequence is mapped to a text embedding vector through the embedding layer of the large language model.

[0010] As a preferred embodiment of the first aspect mentioned above, in S1, when the task type is a multimodal task, the initial embedding sequence is formed by concatenating the image embedding vector and the text embedding vector in the input order, and the image embedding vector is used as the context information vector; when the task type is a long dialogue task, the initial embedding sequence is a text embedding vector, and the text embedding vector representing the historical dialogue is used as the context information vector.

[0011] As a preferred embodiment of the first aspect above, in S2, the anchor word is the word located at the very beginning of the initial embedding sequence. After the anchor word is identified, a parallel dual-track calculation structure consisting of an injection track and a native track is constructed in the multi-head attention module of the Transformer decoding layer to separate the attention calculation paths of the anchor word from those of the other words. The injection track is the attention calculation path of the anchor word, and the native track is the attention calculation path of the other words.

[0012] As a preferred embodiment of the first aspect mentioned above, in the injection track, the causal self-attention in the multi-head attention module is replaced with cross-attention, the hidden state vector of the anchor term is used as the query vector, and the context information vector is used as the key vector and value vector. The new representation of the anchor term is calculated through the cross-attention mechanism.

[0013] As a preferred option for the first aspect mentioned above, the context information vector is pooled before the cross-attention mechanism is calculated to reduce computational complexity.

[0014] Secondly, this invention provides a large language model contextual capability activation system based on attention focus, comprising: The data acquisition module is used to acquire question text or multimodal input data provided by the user, wherein the multimodal input data consists of question text and corresponding images; The result acquisition module is used to generate answers based on the question text or multimodal input data according to the large language model context capability activation method based on attention focus as described in any of the above-mentioned first aspect schemes, and to use the obtained output text as the answer text corresponding to the question text.

[0015] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for stimulating the contextual capabilities of a large language model based on attention points as described in any of the solutions of the first aspect above.

[0016] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the large language model context capability activation method based on attention focus as described in any of the solutions of the first aspect above.

[0017] Compared with the prior art, the present invention has the following advantages: This invention requires no fine-tuning of the large language model; only the attention calculation method during the inference phase needs modification. It can be seamlessly integrated as a plug-and-play module into any existing pre-trained large language model based on the Transformer decoder architecture, activating its contextual capabilities. The contextual information injection operation in this invention only requires one cross-attention calculation, the result of which is stored in a key-value cache for reuse in all subsequent lexical generation steps, thus introducing no additional computational overhead during the autoregressive inference generation phase. Attached Figure Description

[0018] Figure 1 This is a flowchart of the steps of the method of the present invention; Figure 2 This is a schematic diagram of the core calculation process of the method of the present invention; Figure 3 This is a system block diagram of the present invention; Figure 4 This is a schematic diagram of the composition of a computer electronic device according to an embodiment of the present invention. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0020] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0021] This invention proposes a method for stimulating the contextual capabilities of large language models based on attention focus, which is applied to the autoregressive generation process of large language models. For example... Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned method for activating the contextual capabilities of a large language model based on attention points includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0022] S1. Obtain the original input data according to the task requirements, convert the original input data into an initial embedding sequence, and extract the key context information that needs to be continuously monitored in the autoregressive lexical generation process from the initial embedding sequence to obtain the context information vector.

[0023] It should be noted that in S1 of this invention, when the task type is a multimodal task, the original input data consists of images and text, and when the task type is a long dialogue task, the original input data only contains text.

[0024] It should be noted that in S1 of this invention, when the original input data contains an image, the image is segmented and input into the visual encoder to obtain an image embedding vector. When the original input data contains text, the large language model's tokenizer converts the input text into a discrete sequence of text word IDs. The large language model's embedding layer then maps this text word ID sequence into a text embedding vector. .

[0025] In this embodiment, the large language model described above has only a decoder-only structure and no encoder. The image embedding vector is generated by the visual encoder, and the text embedding vector is generated by the first layer of the large language model, namely the embedding layer. Therefore, the method of the present invention can be applied to a multimodal large language model composed of both a visual encoder and a large language model, or it can be applied only to the large language model.

[0026] It should be noted that in S1 of this invention, when the task type is a multimodal task, the initial embedding sequence... It is formed by concatenating the image embedding vector and the text embedding vector in the input order; when the task type is a long dialogue task, the initial embedding sequence is the text embedding vector.

[0027] It should be noted that in S1 of this invention, when the task type is a multimodal task, the image embedding vector is used as the context information vector. When the task type is a long dialogue task, the text representing the historical dialogue is embedded into a vector. As a context information vector .

[0028] Embodiment S1 of this invention aims to preprocess the original input and extract key contextual information. Specifically, existing multimodal tasks in the field of large language models use a combination of images and text as input. The large language model needs to answer based on the provided images and text (questions). In this case, it is necessary to concatenate image embedding vectors and text embedding vectors as the initial embedding sequence, and use the image embedding vector as key contextual information. For long dialogue tasks (text modal tasks), the original input data only contains text. In this case, the initial embedding sequence is the text embedding vector, and only the text embedding vector representing the historical dialogue is used as key contextual information, without the need to concatenate image embedding vectors.

[0029] Furthermore, in S1 of the present invention, an optimization sub-step for key information filtering, retrieval, or enhancement can be added. This sub-step is executed before information injection to improve the quality of the injected information and its relevance to the input text.

[0030] S2. The initial embedding sequence is used as the input to the decoder of the pre-trained large language model. The decoder is composed of multiple Transformer decoding layers cascaded sequentially. Information injection is performed on a specific Transformer decoding layer. The specific processing flow in the Transformer decoding layer that performs information injection is as follows: First, the anchor word in the initial embedding sequence is identified, and the attention calculation paths of the anchor word and the other words are separated in the multi-head attention module of the Transformer decoding layer. In the attention calculation path of the anchor word, the context information vector is injected into the representation of the anchor word through an adaptive query mechanism to obtain a new representation of the anchor word. In the attention calculation path of the other words, the original attention calculation method of the large language model remains unchanged to obtain the representation of the other words. The new representation of the anchor word and the representation of the other words are re-concatenated according to the position of the word in the initial embedding sequence to form an updated sequence representation as the final result of the multi-head attention module of the Transformer decoding layer, thereby completing the processing of the Transformer decoding layer.

[0031] It should be noted that in S2 of this invention, the anchor lexicon corresponds to the predefined Beginning-of-Sequence (BOS) marker in the vocabulary, which is always located at the very beginning of the initial embedding sequence. This lexicon naturally exhibits the characteristic of "attention sinking point" in the Transformer architecture, that is, it continuously obtains a high attention weight during the autoregressive generation process, and is therefore selected as a stable anchor point to carry external context information.

[0032] It should be noted that in S2 of this invention, information injection can be performed on all Transformer decoding layers of the decoder, or information injection can be performed on any number of Transformer decoding layers; no limitation is imposed in this invention. In this embodiment, information injection is performed every 5 Transformer decoding layers, i.e., the 5th Transformer decoding layer, the 10th Transformer decoding layer, the 15th Transformer decoding layer, the 20th Transformer decoding layer, and so on.

[0033] It should be noted that in S2 of this invention, after identifying the anchor word, a parallel dual-track calculation structure consisting of an injection track and a native track is constructed in the Multi-Head Attention (MHA) module of the Transformer decoding layer to separate the attention calculation paths of the anchor word and the other words; wherein, the injection track is the attention calculation path of the anchor word, and the native track is the attention calculation path of the other words.

[0034] It should be noted that in the injection track of S2 in this invention, the causal self-attention in the multi-head attention module is replaced with cross-attention, and the hidden state vector of the anchor word is changed. Using the query vector as the key vector and the context information vector as the value vector, a new representation of the anchor term is calculated through a cross-attention mechanism. .

[0035] To facilitate a better understanding of the present invention by those skilled in the art, the principle of step S2 is described in detail below. In the large language model of this embodiment S2, Several Transformer decoding layers are stacked sequentially to form a decoder. Among them, the first... The Transformer decoding layer receives data from the first layer. Hidden state vector output by the Transformer decoding layer and the hidden state vector Simultaneously, it serves as the input to the multi-head attention module as a query, key, and value vector: in, It is the sequence length. It is the hidden layer dimension; For the first The output of the multi-head attention module in the Transformer decoding layer; Representation layer normalization; This indicates multi-head attention calculation.

[0036] The above describes the processing logic of the original multi-head attention module. This invention improves upon it by designing two parallel computation paths (i.e., the injection path and the native path). Anchor words and other regular words are routed to their respective computation paths for processing. Specifically, as follows... Figure 2 As shown, the injection track (track one) only targets the identified anchor lexical units, replacing their standard causal self-attention with cross-attention. This design allows the anchor lexical units to adaptively retrieve and fuse the most relevant information from external key contextual information, thus transforming them into a high-information-density contextual anchor. The original track (track two) targets all other lexical units except the anchor lexical units, maintaining their original causal self-attention computation mechanism from the pre-trained large language model to obtain the representations of the remaining lexical units. This design aims to fully preserve the original computational flow and learned sequence processing capabilities of the pre-trained large language model, avoiding the introduction of destructive interference. Then, the new representations of the anchor lexical units output from the injection track are concatenated. The remaining lexical representations of the original track output As the final result of the multi-head attention module in the Transformer decoding layer : After obtaining the final result of the above multi-head attention module, Send to the Subsequent submodules within the Transformer decoding layer complete the computation according to the original processing flow of the Transformer decoding layer, thereby obtaining the... Hidden state vector output by the Transformer decoding layer This is then used as the input to the next decoding layer until the forward propagation computation of the entire decoder is completed. Since the processing flow of other sub-modules in the Transformer decoding layer of the large language model is not modified here, the processing logic of other sub-modules will not be elaborated further. In practice, the final result of the multi-head attention module can be used to replace the output of the multi-head attention module before modification, while keeping other processing in the Transformer decoding layer unchanged, thus obtaining the output of the Transformer decoding layer.

[0037] Furthermore, in the technical solution of the present invention, the computational complexity of the cross-attention mechanism in step S2 is... , The length of the query vector is constant and is 1. The length of the key vector or value vector; This refers to the feature dimension, i.e., the hidden layer dimension. If the context information vector is too long (i.e....), (If the length is large), the computation time will increase significantly. To solve this problem, a pooling operation, such as mean pooling, can be performed on the context information vector before cross-attention calculation. The context information vector is compressed to a length of 1, thereby reducing the computational complexity from... Reduce to This improves computational efficiency while preserving key contextual information.

[0038] Furthermore, in order to maximize computational efficiency, the context information injection operation in this invention only needs to perform cross-attention calculation once. This calculation result is stored in the key-value cache (KVcache) inherent in the large language model and reused in all subsequent word generation steps, thereby avoiding the introduction of additional computational overhead and having almost no impact on the reasoning speed of the large language model.

[0039] S3. After processing by all Transformer decoding layers, the final sequence representation output by the decoder is used as the input of the Language Modeling Head (LM-Head) of the large language model to predict the probability distribution of the next lexical. A new lexical is sampled from this probability distribution and appended to the end of the text lexical ID sequence transformed from the input text to form the updated text lexical ID sequence.

[0040] In this embodiment S3, following the above process, after... After processing by one Transformer decoding layer, the final sequence representation is obtained. And it is used as input to the Language Modeling Head (LM-Head) of the large language model, for the last word of the initial embedding sequence. The corresponding final sequence representation Perform linear transformation and softmax operation to predict the next word. probability distribution : in, This represents the language modeling head mentioned above; Indicates by The text consists of all the preceding lexical units. A new lexical unit is sampled from the above probability distribution and appended to the end of the text lexical unit ID sequence to form an updated text lexical unit ID sequence, which is used in the autoregressive lexical unit generation stage inherent in large language models.

[0041] S4. Input the updated text lexical ID sequence into the large language model for autoregressive lexical generation until the complete output text is obtained.

[0042] It should be noted that the S1~S3 processes described above can be considered as the prefill stage before the large language model performs autoregressive lexical generation, i.e., the initial stage of large model inference. This stage is responsible for processing the initial embedding sequence and generating the first output lexical (i.e., the new lexical mentioned above). Its core is to establish a KV cache through parallel computing to support subsequent efficient decoding. In the prefill stage, anchor lexical identification, separation of the attention calculation paths between anchor lexical and other lexical are required, and the concatenation of the new representation of the anchor lexical and the representation of the other lexical is used as the final result of the multi-head attention module. This drives the entire decoder to complete the decoding process, and the language modeling head generates new lexical based on the final sequence representation output by the decoder.

[0043] After the first output lexical generation is completed, the pre-filling stage is considered to be over. In S4, the present invention uses the updated text lexical ID sequence to perform the inherent autoregressive lexical generation of the large language model. That is, the internal embedding layer first remaps the updated text lexical ID sequence into a new text embedding vector. Then, the text embedding vector is further processed by the unmodified decoder (that is, it does not need to identify anchor lexicals, does not need to separate the attention calculation path between anchor lexicals and other lexicals in the multi-head attention module, and does not need to perform cross-attention calculation). Then, the language modeling head generates the next new lexical. The above process is repeated until the autoregressive lexical generation stage ends, and finally the complete output text is obtained.

[0044] To better demonstrate the specific implementation and technical effects of the present invention, the method for stimulating the contextual capabilities of a large language model based on attention points, as shown in steps S1 to S4 of the above preferred implementation, is applied to two specific examples.

[0045] Example The specific implementation process of the large language model contextual capability activation method based on attention focus used in this embodiment is as described above and will not be repeated here.

[0046] As described above, the method of this invention covers both multimodal large language models and large language models that do not include a visual encoder. The effects of applying the method of this invention to the above two types of large language models will be explained below.

[0047] First, this embodiment verifies the application of the method of the present invention in multimodal tasks. Specifically, four mainstream open-source multimodal large language models with different parameter scales and architectures are used, including: Qwen2.5-VL-3B-Instruct, Gemma3-4B-Instruct, Qwen2.5-VL-7B-Instruct, and Gemma3-12B-Instruct. The performance is verified on the RealWorldQA, MMStar, M3CoT, and POPE benchmark sets. RealWorldQA is used to evaluate the model's visual question answering ability in complex real-world scenarios; MMStar is a comprehensive multimodal capability benchmark covering 6 core capabilities and 18 sub-tasks; M3CoT focuses on multimodal, multi-hop, and complex thought chain reasoning. POPE is a benchmark specifically used to quantitatively evaluate object illusion in multimodal models. This embodiment uses Direct and Chain-of-Thought (CoT) as baseline methods for comparison with the present invention, and uses industry-recognized Accuracy and Macro-F1 as the main evaluation metrics.

[0048] The experimental results are shown in Table 1. The results demonstrate that, across four datasets and four base models, the method described in this invention significantly outperforms the Direct and CoT baseline methods in the vast majority of test scenarios. Taking the most representative Qwen2.5-VL-7B-Instruct model as an example, on the M3CoT dataset, the accuracy of this invention (66.78%) is 22.95% higher than the CoT baseline (43.83%); on the POPE dataset, which specifically detects hallucinations, the accuracy of this invention (85.51%) is also higher than the CoT baseline (80.36%). This result verifies the effectiveness of this invention in enhancing the contextual capabilities of multimodal large language models.

[0049] Table 1. Test results of the multimodal large language model on multimodal tasks. Secondly, this embodiment verifies the application of the method of the present invention in plain text tasks. Specifically, three mainstream open-source large language models are used, including: MiniCPM3-4B, Qwen2.5-7B-Instruct, and Llama3.1-8B-Instruct. Two classic text question answering benchmark datasets are also selected: QuAC (and its subsets divided by dialogue rounds) and SQuAD2.0.

[0050] The experimental results are shown in Table 2. The results demonstrate that the method described in this invention outperforms the Direct and CoT baselines in almost all text task test scenarios. The advantages of this invention are particularly evident on the QuAC dataset. As the context length increases, the performance of the baseline methods declines significantly, exhibiting substantial context forgetting. In contrast, the method described in this invention maintains high performance more effectively. Taking the Llama3.1-8B-Instruct model under the longest context setting as an example, the accuracy of this invention (53.60%) is significantly improved by 18.88% compared to the CoT baseline (34.72%). This result verifies the effectiveness of this invention in enhancing the contextual capabilities of large language models.

[0051] Table 2. Test results of the large language model on pure text tasks It should also be noted that the attention-based method for stimulating the contextual capabilities of a large language model in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a system for stimulating the contextual capabilities of a large language model based on attention points, corresponding to the attention-based method for stimulating the contextual capabilities of a large language model provided in the above embodiments, such as... Figure 3 As shown, it includes: The data acquisition module is used to acquire question text or multimodal input data provided by the user, wherein the multimodal input data consists of question text and corresponding images; The result acquisition module is used to generate answers based on the question text or multimodal input data according to the large language model context capability activation method based on attention focus described in the above embodiments, and to use the obtained output text as the answer text corresponding to the question text.

[0052] It is understood that the large language model contextual ability activation method based on attention focus described in S1-S4 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the large language model contextual ability activation method based on attention focus provided in the above embodiments, such as... Figure 4 As shown, it includes a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the large language model context capability activation method based on attention focus in the above embodiments when executing the computer program.

[0053] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0054] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the large language model context capability activation method based on attention focus provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the large language model context capability activation method based on attention focus in the above embodiments.

[0055] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0056] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0057] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0058] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.< / bos>

Claims

1. A method for stimulating context capabilities of a large language model based on attention sink points, characterized in that, The method comprises the following steps: S1, obtaining original input data according to task requirements, converting the original input data into an initial embedding sequence, extracting key context information that needs to be continuously focused on in the autoregressive token generation process from the initial embedding sequence to obtain a context information vector; S2, inputting the initial embedding sequence as a decoder of a pre-trained large language model, the decoder being sequentially cascaded by a plurality of Transformer decoding layers, and performing information injection on a specific Transformer decoding layer; the specific processing procedure in the Transformer decoding layer performing information injection is as follows: first, identifying anchor tokens in the initial embedding sequence, and separating the attention calculation paths of the anchor tokens and the remaining tokens in the multi-head attention module of the Transformer decoding layer; in the attention calculation path of the anchor tokens, the context information vector is injected into the representation of the anchor tokens through an adaptive query mechanism to obtain a new representation of the anchor tokens; in the attention calculation path of the remaining tokens, the original attention calculation mode of the large language model is maintained unchanged to obtain the representation of the remaining tokens; the new representation of the anchor tokens and the representation of the remaining tokens are re-spliced according to the positions of the tokens in the initial embedding sequence to form an updated sequence representation as the final result of the multi-head attention module of the Transformer decoding layer, thereby completing the processing of the Transformer decoding layer; S3, after processing by all the Transformer decoding layers, inputting the final sequence representation output by the decoder as an input of a language modeling head of the large language model to predict a probability distribution of the next token, sampling a new token from the probability distribution and appending the new token to the end of the input text token ID sequence converted from the text to form an updated text token ID sequence; S4, inputting the updated text token ID sequence into the large language model for autoregressive token generation until a complete output text is obtained.

2. The attention sink-based large language model context capability activation method of claim 1, wherein, In S1, when the task type is a multi-modal task, the original input data consists of an image and text, and when the task type is a long dialogue task, the original input data only contains text.

3. The attention sink-based large language model context capability activation method of claim 2, wherein, In S1, when the original input data contains an image, the image is input into a visual encoder after being divided into blocks to obtain an image embedding vector; when the original input data contains text, the input text is converted into a discrete text token ID sequence by using a tokenizer of the large language model, and the text token ID sequence is mapped into a text embedding vector by an embedding layer of the large language model.

4. The attention sink-based large language model context capability activation method of claim 3, wherein, In S1, when the task type is a multi-modal task, the initial embedding sequence is formed by splicing the image embedding vector and the text embedding vector in the input order, and the image embedding vector is taken as the context information vector; when the task type is a long dialogue task, the initial embedding sequence is the text embedding vector, and the text embedding vector representing the historical dialogue is taken as the context information vector.

5. The attention sink-based large language model context capability activation method of claim 1, wherein, In S2, the anchor token is the token located at the front end of the initial embedding sequence. After identifying the anchor token, a parallel double-track computing structure composed of an injection track and a native track is constructed in the multi-head attention module of the Transformer decoding layer to separate the attention calculation paths of the anchor token and the remaining tokens; wherein the injection track is the attention calculation path of the anchor token, and the native track is the attention calculation path of the remaining tokens.

6. The attention sink-based large language model context capability activation method of claim 5, wherein, In the injection track, the causal self-attention in the multi-head attention module is replaced by cross-attention, the hidden state vector of the anchor token is taken as the query vector, and the context information vector is taken as the key vector and the value vector, and the new representation of the anchor token is calculated through the cross-attention mechanism.

7. The attention sink-based large language model context capability activation method of claim 6, wherein, Before performing the cross-attention mechanism calculation, the context information vector is subjected to a pooling operation to reduce the computational complexity.

8. An attention sink-based large language model context capability activation system, characterized in that, The method comprises the following steps: A data acquisition module is configured to acquire a question text or multi-modal input data provided by a user, wherein the multi-modal input data is composed of the question text and a corresponding image. A result acquisition module is configured to perform answer generation based on the question text or the multi-modal input data according to the large language model context ability excitation method based on an attention sink point according to any one of claims 1 to 7, and obtain an output text as an answer text corresponding to the question text.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by the processor, the large language model context ability excitation method based on an attention sink point according to any one of claims 1 to 7 is implemented.

10. A computer electronic device, comprising: The device comprises a memory and a processor. The memory is configured to store a computer program. The processor is configured to implement the large language model context ability excitation method based on an attention sink point according to any one of claims 1 to 7 when the computer program is executed.

Images