Large language model attention dynamic feature extraction method based on syntactic prior constraint

Abstract

The application discloses a large language model attention dynamic feature extraction method based on syntax prior constraint and belongs to the natural language processing field. The method is applied to a large language model attention module and comprises the following steps: obtaining a target text and Token of the target text, performing dependency syntax analysis on the target text to determine dependency relation labels of each Token, and calculating a prior vector for syntax perception; obtaining attention column aggregation vectors of each attention head of each layer of the large language model to calculate total attention degrees of each Token by each attention head and alignment scores containing exponential decay penalty terms to quantify calculation effectiveness of each attention head; introducing a temperature coefficient to calculate a refined matrix used as an output of the attention module based on the attention column aggregation vectors of each attention head and the alignment scores. The scheme can improve logicality and structural explainability of an extraction result. In addition, the addition of the exponential decay penalty terms can greatly improve a denoising effect.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for extracting dynamic features of attention from a large language model based on syntactic prior constraints. Background Technology

[0002] With the development of deep learning technology, large language models based on the Transformer architecture (such as BERT, RoBERTa, LLaMA, etc.) have achieved great success in natural language processing tasks. The core of these models lies in the multi-head self-attention mechanism, which aims to capture the correlation between different tokens in the input sequence through multiple attention heads.

[0003] However, in practical applications, especially in tasks requiring high-precision extraction of the core skeleton of text (such as saliency detection and keyword extraction), directly using the original attention matrix of a large language model (LLM) suffers from high noise and a lack of syntactic awareness. In LLM, many attention heads fail to capture meaningful semantics, instead focusing excessively on punctuation, special placeholders, or simply themselves. These noisy heads severely interfere with the computation of the core semantics of the text.

[0004] Existing attention filtering methods typically use static binary masks (i.e., only nouns are set to 1, and the rest to 0) to filter attention heads. This hard-constraint method has serious flaws when processing Chinese because it cannot distinguish the weight differences between core predicates, logical connectors, and ordinary nouns in the sentence structure. This results in loosely structured and illogical extraction results, leading to a lack of syntactic awareness.

[0005] Therefore, there is an urgent need to provide a method for dynamic feature extraction of attention in large language models based on syntactic prior constraints. Summary of the Invention

[0006] To address the issues of high noise and lack of syntactic awareness in the original attention module of traditional large language models (LLMs), this invention provides a method for dynamic feature extraction of attention in large language models based on syntactic prior constraints.

[0007] On the one hand, a dynamic feature extraction method for attention in large language models based on syntactic prior constraints is provided, which is applied to the attention module of large language models. The method includes: Obtain the target text and the token of the target text, perform dependency parsing on the target text to determine the dependency relation label of each token, and calculate the prior vector for syntax awareness; Obtain the attention column aggregation vector of each attention head in each layer of the large language model, and combine the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, so as to quantify the computational effectiveness of each attention head. Based on the attention column aggregation vector of each attention head and the alignment score, a temperature coefficient is introduced to calculate the refinement matrix used as the output of the attention module.

[0008] On the other hand, a dynamic feature extraction device for attention of a large language model based on syntactic prior constraints, based on the steps described in any method embodiment of the specification, is provided. This device is configured within the attention module of the large language model and includes: The masking unit is used to obtain the target text and the token of the target text, perform dependency parsing on the target text to determine the dependency relationship label of each token, and calculate the prior vector for syntax awareness. The metric unit is used to obtain the attention column aggregation vector of each attention head in each layer of the large language model, so as to combine the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, so as to quantify the computational effectiveness of each attention head. The fusion unit is used to aggregate the attention column vectors of each attention head and the alignment score, and to introduce a temperature coefficient to calculate a refined matrix as the output of the attention module.

[0009] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for executing the computer program stored in the memory to implement the steps of the method described above.

[0010] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the method described above.

[0011] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0012] The technical solution provided by this invention can bring at least the following beneficial effects: This paper replaces the original {0, 1} mask with a syntax-aware prior vector to dynamically measure and filter high-quality attention heads in LLM that understand both syntax and semantics. Compared to static binary masks (i.e., only keeping nouns as 1 and the rest as 0) to filter attention heads, the syntax-aware prior vector in this scheme can distinguish the weight differences between the core predicate (ROOT), logical connectors (mark), and ordinary nouns in the sentence structure, which can improve the logicality and structural interpretability of the extraction results. In addition, when dealing with complex texts containing a lot of punctuation and long sentences, by adding an exponential decay penalty term, it can dynamically remove attention noise from large models compared to the baseline method without a penalty term. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a method for dynamic feature extraction of attention from a large language model based on syntactic prior constraints, provided by an embodiment of the present invention. Figure 2 This is a structural diagram of a large language model attention dynamic feature extraction device based on syntactic prior constraints provided in an embodiment of the present invention; Figure 3 This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0016] The following describes the specific implementation of the above concept.

[0017] Please refer to Figure 1 This invention provides a method for dynamic feature extraction of attention in a large language model based on syntactic prior constraints, applied to the attention module of a large language model. The method includes: Step 100: Obtain the target text and the target text's token, perform dependency parsing on the target text to determine the dependency relation label for each token, and calculate the prior vector for syntax awareness; Step 102: Obtain the attention column aggregation vector of each attention head in each layer of the large language model, and combine it with the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, so as to quantify the computational effectiveness of each attention head. Step 104: Based on the attention column aggregation vector and alignment score of each attention head, a temperature coefficient is introduced to calculate the refined matrix used as the output of the attention module.

[0018] In this embodiment of the invention, a syntax-aware prior vector is used instead of the original {0, 1} mask. This allows for the dynamic measurement and selection of high-quality attention heads in the LLM that understand both syntax and semantics. Compared to static binary masks (i.e., only nouns are set to 1, and the rest to 0) for filtering attention heads, the syntax-aware prior vector in this scheme can distinguish the weight differences between the core predicate (ROOT), logical connectors (mark), and ordinary nouns in the sentence structure, thereby improving the logicality and structural interpretability of the extraction results. Furthermore, when processing complex texts containing numerous punctuation marks and long sentences, by adding an exponential decay penalty term, compared to the baseline method without a penalty term, large model attention noise can be dynamically removed.

[0019] The following description Figure 1 The execution method for each step is shown.

[0020] For step 100: It should be noted that this solution is an improvement on the attention module of a large language model. A feature extraction module preceding the large model can output a token for the target text. This step 100 is executed in parallel with the original attention matrix extraction algorithm of the large language model's attention module.

[0021] Unlike existing technologies that use discrete masks of {0, 1}, this embodiment constructs a continuous numerical syntactic prior vector. First, dependency parsing is performed on the target text to obtain each token. The dependency relationship labels (deprel) are then used to generate prior vectors through a multi-level weighted mapping function.

[0022] In some implementations, the prior vector is calculated as follows: In the formula, Let i be the i-th element of the prior vector. The preset topological importance weights, This indicates whether the tag of the token is a root word; if so, set it to 1, otherwise set it to 0. A set of logical connectives, This indicates whether the tag of the token is a logical conjunction; if yes, set it to 1, otherwise set it to 0. Represents a set of entity nouns. This indicates whether the tag of the token is an entity noun; if yes, set it to 1, otherwise set it to 0. This is a non-zero smoothing term used to preserve the weak interest of background words.

[0023] in: ={mark, case, ccomp} represents a set of logical connectives; ={nsubj, obj,iobj} represents the set of entity nouns; The preset topological importance weights are used, and they satisfy the following conditions: (In this embodiment, preferably) =1.0, =0.9, =0.8), reflecting the "predicate core". Logical connection The hierarchical structure of "entity participants". Non-zero smoothing terms (e.g., 0.1) are used to preserve the subtle attention to background words. This makes the attention heatmap extracted by this method, i.e., the refined matrix, highly consistent with human linguistic intuition, and can clearly present the skeletal structure of "predicate-logic-entity".

[0024] Regarding step 102: In some implementations, step 102 may include: Obtain the aggregated attention column vectors of each attention head in each layer of the large language model; Based on the attention column aggregation vector, calculate the total attention of the attention head to each token; The total attention of the attention head is calculated using cosine similarity to determine the degree of matching between the total attention and the prior vector, in order to calculate semantic and structural consistency. A punctuation noise penalty term and a self-loop redundancy penalty term are introduced to calculate the alignment score of the attention head.

[0025] It should be noted that the original attention matrix extraction algorithm of the large language model outputs an attention matrix, which contains the aggregated vector of attention columns of each attention head in each layer. In this embodiment, in order to quantify whether a certain attention head in the LLM is "effective", an alignment score S(l,h) with a noise penalty mechanism is defined.

[0026] In some implementations, the alignment score is calculated as follows: in, In the formula, It is the alignment score of the h-th attention head in the l-th layer. The total attention of the h-th attention head in the l-th layer. For the prior vector, This is the punctuation penalty coefficient. The percentage of attention weight assigned to the h-th attention head in layer l for the punctuation mark. The weight percentages of the diagonal elements of the attention matrix. The self-loop penalty coefficient, Let N be the attention column aggregation vector of the h-th attention head in the l-th layer, and N be the number of tokens.

[0027] In this embodiment, cosine similarity is used to calculate the matching degree between the total attention of the attention head and the prior vector to compute semantic and structural consistency. Two exponential decay penalty terms are introduced: a punctuation noise penalty term and a self-loop redundancy penalty term. When an attention head overemphasizes punctuation, the punctuation noise penalty term rapidly approaches 0, thereby suppressing the score of that attention head. The self-loop redundancy penalty term is used to eliminate attention heads that only perform simple word embedding mapping without contextual interaction. By introducing these penalty terms, when processing complex texts containing a large number of punctuation marks and long sentences, the F1 score can be improved by 5%-10% compared to the baseline method without penalty terms.

[0028] Regarding step 104: In some implementations, step 104 may include: Based on the alignment scores of each attention head, select the K attention heads with the highest alignment scores; The weight of each selected attention head is calculated based on the alignment score and temperature coefficient of the selected K attention heads. Based on the weights of each selected attention head and the attention column aggregation vector, a refined matrix is ​​calculated to be used as the output of the attention module.

[0029] In this embodiment, after calculating all L After scoring H heads, the system selects the top-K heads with the highest scores. To avoid the simple averaging operation diluting the contribution of the best head, a temperature-aware Softmax weighted fusion algorithm is used to generate the final refined matrix as the output of the attention module. Traditional attention modules directly output the attention matrix extracted by the original attention extraction algorithm. This approach constructs explicit syntactic constraints, utilizes the prior vectors of these constraints and the attention matrix output by the traditional attention module, performs alignment metrics and penalty operations to select attention heads that understand both syntax and semantics. Finally, temperature-aware weighted fusion yields the refined matrix as the output of the attention module, replacing the original attention matrix. This process requires no parameter fine-tuning of the LLM (Parameter-Efficient), only forward inference and matrix operations, making it computationally efficient and easy to deploy quickly.

[0030] Specifically, the refinement matrix is ​​calculated as follows: in, In the formula, To refine the matrix, Let h be the weight of the attention head in the l-th layer. This is the attention column aggregation vector of the h-th attention head in the l-th layer. The alignment score for the selected attention head. This is the temperature coefficient.

[0031] when When →0, the fusion result tends to retain only the strongest head, suitable for high-precision scenarios; when → At that time, it tends towards average fusion. This invention adjusts... It achieves a dynamic balance between "focus" and "smoothing".

[0032] Therefore, this solution can combine explicit syntactic knowledge to dynamically remove attention noise from large models and accurately extract semantic skeletons.

[0033] Please refer to Figure 2 This invention provides a dynamic feature extraction device for attention in a large language model based on syntactic prior constraints. The device is configured within the attention module of the large language model and is used to implement the steps of any method embodiment described in the specification. The device includes: Masking unit 201 is used to obtain the target text and the token of the target text, perform dependency parsing on the target text to determine the dependency relationship label of each token, and calculate the prior vector for syntax awareness. The metric unit 202 is used to obtain the attention column aggregation vector of each attention head in each layer of the large language model, so as to combine the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, so as to quantify the computational effectiveness of each attention head. The fusion unit 203 is used to aggregate the attention column vector and alignment score based on each attention head, and introduces a temperature coefficient to calculate the refined matrix as the output of the attention module.

[0034] It should be noted that the above-described device embodiments and method embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0035] Embodiments of this application also provide a computer device, please refer to... Figure 3 The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, wherein at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the large language model attention dynamic feature extraction method based on syntactic prior constraints provided in the above-described method embodiments.

[0036] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the large language model attention dynamic feature extraction method based on syntactic prior constraints provided in the above-described method embodiments.

[0037] Embodiments of this application also provide a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium and executes the computer program, causing the computer device to perform any of the large language model attention dynamic feature extraction methods based on syntactic prior constraints described in the above embodiments.

[0038] For ease of description, the above devices or apparatuses are described separately according to their functions, divided into various modules or units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0039] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0040] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0041] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for dynamic feature extraction of attention in a large language model based on syntactic prior constraints, applied to the attention module of a large language model, characterized in that, include: Obtain the target text and the token of the target text, perform dependency parsing on the target text to determine the dependency relation label of each token, and calculate the prior vector for syntax awareness; Obtain the attention column aggregation vector of each attention head in each layer of the large language model, and combine the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, so as to quantify the computational effectiveness of each attention head. Based on the attention column aggregation vector of each attention head and the alignment score, a temperature coefficient is introduced to calculate the refinement matrix used as the output of the attention module.

2. The method as described in claim 1, characterized in that, The prior vector is calculated as follows: In the formula, Let i be the i-th element of the prior vector. The preset topological importance weights, This indicates whether the tag of the token is a root word; if so, set it to 1, otherwise set it to 0. It is a set of logical connectives. This indicates whether the tag of the token is a logical conjunction; if yes, set it to 1, otherwise set it to 0. Represents a set of entity nouns. This indicates whether the tag of the token is an entity noun; if yes, set it to 1, otherwise set it to 0. This is a non-zero smoothing term used to preserve the weak interest of background words.

3. The method as described in claim 1, characterized in that, The step of obtaining the attention column aggregation vector of each attention head in each layer of the large language model, and combining it with the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, includes: Obtain the aggregated attention column vectors of each attention head in each layer of the large language model; Based on the attention column aggregation vector, calculate the total attention of the attention head for each token; The total attention of the attention head is calculated using cosine similarity to determine the degree of matching between the total attention and the prior vector, in order to calculate semantic and structural consistency. A punctuation noise penalty term and a self-loop redundancy penalty term are introduced to calculate the alignment score of the attention head.

4. The method as described in claim 3, characterized in that, The alignment score is calculated as follows: in, In the formula, It is the alignment score of the h-th attention head in the l-th layer. The total attention of the h-th attention head in the l-th layer. Let be the prior vector. This is the punctuation penalty coefficient. The percentage of attention weights assigned to the h-th attention head in layer l for the punctuation mark. The weight percentages of the diagonal elements of the attention matrix. The self-loop penalty coefficient, Let N be the attention column aggregation vector of the h-th attention head in the l-th layer, and N be the number of tokens.

5. The method as described in claim 1, characterized in that, The attention column aggregation vector based on each attention head and the alignment score, combined with a temperature coefficient, are used to calculate a refined matrix as the output of the attention module, including: Based on the alignment score of each attention head, select the K attention heads with the highest alignment scores; The weight of each selected attention head is calculated based on the alignment score and temperature coefficient of the selected K attention heads. Based on the weights of each selected attention head and the attention column aggregation vector, a refined matrix is ​​calculated to be used as the output of the attention module.

6. The method as described in claim 5, characterized in that, The refining matrix is ​​calculated as follows: in, In the formula, To refine the matrix, Let h be the weight of the attention head in the l-th layer. This is the attention column aggregation vector of the h-th attention head in the l-th layer. The alignment score for the selected attention head. This is the temperature coefficient.

7. A dynamic feature extraction device for attention in a large language model based on syntactic prior constraints, configured in the attention module of the large language model, for implementing the steps of the method described in any one of claims 1-6, characterized in that, include: The masking unit is used to obtain the target text and the token of the target text, perform dependency parsing on the target text to determine the dependency relationship label of each token, and calculate the prior vector for syntax awareness. The metric unit is used to obtain the attention column aggregation vector of each attention head in each layer of the large language model, so as to combine the prior vector to calculate the total attention of each attention head to each token and the alignment score containing the exponential decay penalty term, so as to quantify the computational effectiveness of each attention head. The fusion unit is used to aggregate the attention column vectors of each attention head and the alignment score, and to introduce a temperature coefficient to calculate a refined matrix as the output of the attention module.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images