Short answer scoring method and system based on fusion of hierarchical attention and dynamic gating

Abstract

The invention discloses a short answer scoring method and system based on hierarchical attention and dynamic gating fusion, and belongs to the field of natural language processing and education intellectualization, and the method comprises the steps: splicing a reference answer text and a student answer text into a structured input sequence, and converting the structured input sequence into a feature vector; processing the feature vectors through a block context awareness attention mechanism, and capturing semantic dependence in local blocks and cross-block semantic interaction; adaptively fusing semantic dependence in local blocks and cross-block semantic interaction by adopting a dynamic gating mechanism to obtain fused multi-level semantic representation; modeling global long-range semantic association through a superposed multi-head self-attention layer, and compensating semantic omission of a blocking mechanism; and based on the fused multi-level semantic representation, calculating the similarity between the reference answer and the student answering text, and outputting a scoring result. According to the method, the problem of unbalance between local focusing and global coherence in single-stage fusion is solved through a local-cross-block-global three-stage attention architecture.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and intelligent education technology, and in particular relates to a short answer scoring method and system based on the fusion of hierarchical attention and dynamic gating. Background Art

[0002] In the field of automatic scoring of short answers, traditional methods are mainly divided into rule-based methods, traditional machine learning methods, and deep learning-based pre-trained models. Rule-based methods evaluate answer similarity by manually designing matching patterns, which is highly interpretable in specific domains. Traditional machine learning-based models rely on manually constructed features, such as word frequency, TF-IDF, and word embeddings, to capture the basic semantic information of the text. Deep learning-based pre-trained models utilize architectures such as BERT to automatically learn semantic representations and model text associations through global attention mechanisms.

[0003] However, there are significant defects in the existing technology: first, rule-based methods and traditional machine learning models have difficulty handling non-standard expressions and implicit logical reasoning in students' answers, and are insensitive to changes in local contextual semantics; second, the homogenized global attention mechanism of the pre-trained model causes irrelevant vocabulary to interfere with key semantic units, and the single-layer feature fusion structure cannot balance local fine-grained matching and global coherence, resulting in the failure of cross-level semantic association modeling, which ultimately restricts the accuracy of scoring. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a short answer scoring method and system based on the fusion of hierarchical attention and dynamic gating to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a short answer scoring method based on the fusion of hierarchical attention and dynamic gating, comprising:

[0006] Concatenate the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector;

[0007] Processing the feature vectors through a block-wise context-aware attention mechanism to capture local intra-block semantic dependencies and cross-block semantic interactions;

[0008] A dynamic gating mechanism is used to adaptively fuse the local intra-block semantic dependency and cross-block semantic interaction to obtain a fused multi-level semantic representation;

[0009] The stacked multi-head self-attention layer models global long-range semantic associations to compensate for semantic omissions in the chunking mechanism.

[0010] Based on the fused multi-level semantic representation, the similarity between the reference answer and the student's answer text is calculated, and the scoring result is output.

[0011] Preferably, the formula for concatenating the reference answer text and the student answer text into a structured input sequence is:

[0012]

[0013] Among them, [CLS] is the classification mark, R is the reference answer, [SEP] is the separator, and S is the student answer. Indicates sequence splicing.

[0014] Preferably, the block-wise context-aware attention mechanism includes:

[0015] Divide the input sequence into K consecutive semantic blocks;

[0016] Calculate the scaled dot product attention within each semantic block to generate intra-block attention features;

[0017] A sliding window mechanism is used to make the current block interact with the previous j historical blocks across blocks to generate cross-block attention features.

[0018] Preferably, the calculation formula for the cross-block attention interaction is:

[0019]

[0020] Among them, h is the number of attention heads, is the scaling factor, d is the embedding dimension, Q i is the query vector of the i-th block, is the keyword vector of the first j blocks, V j is the value vector of the first j blocks.

[0021] Preferably, the dynamic gating mechanism includes:

[0022] Generate dimension-sensitive gating weight matrix through Sigmoid function;

[0023] According to formula A i =G i ⊙(A intra +A inter ) weighted fusion of intra-block attention features and cross-block attention features; where ⊙ represents element-by-element multiplication, A i is the result of the gated fusion of the i-th block, G i is the weight of the gated fusion of the i-th block, is the result of self-attention calculation within the i-th block, is the attention interaction result between the i-th block and the previous i blocks.

[0024] Preferably, the generation formula of the gating weight matrix is:

[0025] F=σ(W g ·RMSNorm(H0));

[0026] Among them, σ is the Sigmoid function, W g is a learnable parameter, RMSNorm is root mean square normalization, and H0 is the context-aware embedding matrix.

[0027] Preferably, the process of modeling global long-range semantic associations includes:

[0028] The semantic association between any positions in the entire sequence is calculated through the standard Softmax attention layer;

[0029] The output is fused with the block attention features through residual connection and RMSNorm layer to generate global enhanced features.

[0030] Preferably, the steps of calculating the similarity between the reference answer and the student's answer text and outputting the scoring result include:

[0031] Perform mean pooling on the global enhanced features after dimensionality reduction through a fully connected layer;

[0032] The binary logits are output through the linear transformation layer, and the cross entropy loss function is applied to train the model.

[0033] In a second aspect, the present invention further discloses a short answer scoring system based on the fusion of hierarchical attention and dynamic gating, which is used to implement the method described in the first aspect. The system includes:

[0034] Input processing module, used to splice the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector;

[0035] A block attention module is used to process the feature vector through a block context-aware attention mechanism to capture the semantic dependencies within the local block and the semantic interactions across blocks;

[0036] A dynamic gating module, configured to adaptively fuse the local intra-block semantic dependency and cross-block semantic interaction using a dynamic gating mechanism to obtain a fused multi-level semantic representation;

[0037] The global attention module is used to model global long-range semantic associations through stacked multi-head self-attention layers to compensate for semantic omissions in the chunking mechanism;

[0038] The scoring output module is used to calculate the similarity between the reference answer and the student's answer text based on the fused multi-level semantic representation, and output the scoring result.

[0039] In a third aspect, the present invention further discloses a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0040] Compared with the prior art, the present invention has the following advantages and technical effects:

[0041] The present invention provides a short answer scoring method based on the fusion of hierarchical attention and dynamic gating, comprising: first, splicing the reference answer text and the student answer text into a structured input sequence and converting it into a feature vector; second, processing the feature vector through a block context-aware attention mechanism to capture local intra-block semantic dependencies and cross-block semantic interactions; then, using a dynamic gating mechanism to adaptively fuse the local intra-block semantic dependencies and cross-block semantic interactions to obtain a fused multi-level semantic representation; third, modeling global long-range semantic associations through superimposed multi-head self-attention layers to compensate for the semantic omissions of the block mechanism; finally, based on the fused multi-level semantic representation, calculating the similarity between the reference answer and the student answer text, and outputting the scoring result.

[0042] The present invention explicitly divides semantic units through a block-based context-aware attention mechanism. Under the dual effects of intra-block self-attention and cross-block sliding window interaction, it effectively analyzes the non-standard expressions and implicit reasoning logic in students' answers, and improves sensitivity to local subtle semantic changes.

[0043] The present invention adaptively adjusts the fusion weights of local block features and cross-block interaction features based on a dynamic gating mechanism, suppresses noise interference and enhances key semantic discrimination, filters irrelevant vocabulary interference, and strengthens the representation ability of discriminative semantic units.

[0044] The present invention adopts a three-level attention architecture of "local-cross-block-global". The block mechanism focuses on fine-grained matching, achieves balanced modeling of cross-level semantics, and uses global Softmax attention to compensate for long-range dependencies, thus solving the imbalance problem between local focus and global coherence in single-stage fusion.

[0045] The present invention calculates similarity by fusing optimized multi-level semantic representations, significantly improving the accuracy of parsing cross-level semantic associations in fragmented texts, ensuring the reliability of scoring in complex contexts, and improving the accuracy and robustness of scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0047] Figure 1 This is a flow chart of a short answer scoring method based on the fusion of hierarchical attention and dynamic gating according to an embodiment of the present invention;

[0048] Figure 2 This is a cross-block interaction operation diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] Example 1

[0052] like Figure 1 As shown, this embodiment provides a short answer scoring method based on the fusion of hierarchical attention and dynamic gating, including:

[0053] S1. Concatenate the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector;

[0054] Input sequence concatenation: Concatenate the reference answer (R) and the student answer (S) into a structured input sequence in the following format:

[0055]

[0056] Among them, [CLS] is the classification mark, [SEP] is the separator, Indicates sequence concatenation. For example, if the reference answer is "Photosynthesis requires light" and the student's answer is "Plants perform photosynthesis in sunlight", the concatenated input is:

[0057] [CLS]Photosynthesis requires light[SEP]Plants carry out photosynthesis in sunlight[SEP].

[0058] S2. Processing the feature vector through a block-wise context-aware attention mechanism to capture the semantic dependencies within the local block and the semantic interactions across blocks;

[0059] Contextual embedding generation: The pre-trained DeBERTa-v3 model (embedding dimension d = 768) is used to encode the input sequence and output a context-aware embedding matrix:

[0060]

[0061] Among them, L is the input sequence length (maximum supported 512 tokens), and d=768 is the embedding dimension.

[0062] The block-wise context-aware attention mechanism captures local and cross-block semantic dependencies, specifically:

[0063] Chunking strategy: Divide the input sequence into The block length is B = 8, and the sliding window step size is the same as the block length. If there are less than B tokens at the end, they are padded with zeros. For example, when L = 100, it is divided into 13 blocks (12 full blocks + 1 padded block).

[0064] Projection calculation: For each block Perform linear projection and activation:

[0065] Q i ,K i ,V i =SiLU(W q H i ),SiLU(W k H i ),SiLU(W v H i ) (3)

[0066] Among them, Q i is the query vector of the i-th block, K i is the health vector of the i-th block, V i is the value vector of the i-th block, the right subscript i is the i-th block, Q i ,K i ,V i Learnable parameters; the SiLU activation function is defined as SiLU(x) = x·σ(x), which is the Sigmoid linear activation function; is the input of the i-th block, B is the block length, and d is the embedding dimension.

[0067] Intra-block and cross-block attention calculation:

[0068] Intra-block attention: Calculates the attention relationship between tokens within the current block:

[0069]

[0070] Among them, h=8 is the number of attention heads, is the scaling factor, d is the embedding dimension, Q i is the query vector of the i-th block, is the keyword vector of the first i blocks, V i is the value vector of the first i blocks.

[0071] Inter-block attention: To enhance the continuity of local semantics and avoid information fragmentation, a sliding window mechanism is used. The sliding window step size is set to be exactly equal to B. If the end of the sequence is less than a block length, it is padded with zeros to B tokens. Figure 2 A diagram of cross-block interaction operations is shown; Figure 2 (a) in the figure is the global interaction diagram for calculating the first two historical blocks and the current second block. Figure 2 (b) in the figure is the global interaction diagram for calculating the first three historical blocks and the current third block. Figure 2 (c) in the figure is the global interaction graph for calculating the first n historical blocks and the current n-th block.

[0072] The global interaction formula for calculating the previous j historical blocks and the current i-th block is:

[0073]

[0074] Where j<=i, i>1, h=8 is the number of attention heads, is the scaling factor, d is the embedding dimension, Q i is the query vector of the i-th block, is the keyword vector of the first j blocks, V j is the value vector of the first j blocks.

[0075] In this embodiment, through cross-block interaction, the model can capture the continuity of local semantics and avoid information fragmentation.

[0076] S3, using a dynamic gating mechanism to adaptively fuse the local intra-block semantic dependency and cross-block semantic interaction to obtain a fused multi-level semantic representation;

[0077] Furthermore, local and cross-block attention are fused and the output is modulated by gated weights;

[0078] Dynamic gated fusion: Generate gating weight matrix G∈[0,1] L×d , adaptively adjust the feature fusion strength:

[0079] G=σ(W g ·RMSNorm(H0))∈[0,1] L×d (6)

[0080] Among them, σ is the Sigmoid function, W g is a learnable parameter, RMSNorm is root mean square normalization, and H0 is the context-aware embedding matrix.

[0081] Finally, the block attention output is weighted by gating:

[0082]

[0083] Among them, ⊙ represents element-by-element multiplication, A i is the result of the gated fusion of the i-th block, G i is the weight of the gated fusion of the i-th block, is the result of self-attention calculation within the i-th block, is the attention interaction result between the i-th block and the previous i blocks.

[0084] In this embodiment, residual connections and RMSNorm are introduced to alleviate the vanishing gradient problem in deep network training. After concatenating the block attention outputs, residual connections and RMSNorm are used to stabilize the training:

[0085] H1=W o Concat(A1,…,A K ) (8)

[0086] H2=RMSNorm(H0+H1) (9)

[0087] Among them, W o is the output projection matrix.

[0088] S4, modeling global long-range semantic associations through stacked multi-head self-attention layers to compensate for semantic omissions in the chunking mechanism;

[0089] Specifically, a stacked multi-head self-attention layer is used to establish long-range semantic associations between any positions in the entire sequence, including:

[0090] To supplement the long-range dependencies that may be missed by the block mechanism, a standard multi-head Softmax attention layer is added to repair the global correlations that may be missed by the block mechanism:

[0091] Q'=W q 'H2, K'=W k 'H2, V'=W v 'H2 (10)

[0092]

[0093] Among them, d h =d / h=96, number of attention heads h=8.

[0094] The output is connected and normalized via residual connection:

[0095] H3=RMSNorm(H2+A') (12)

[0096] This layer explicitly models the semantic associations between any positions in the entire sequence, complementing the block attention.

[0097] S5. Based on the fused multi-level semantic representation, calculate the similarity between the reference answer and the student's answer text, and output the scoring result.

[0098] Furthermore, the steps of calculating the similarity between the reference answer and the student's answer text and outputting the scoring result include:

[0099] S501, performing mean pooling on the global enhanced features after dimensionality reduction via a fully connected layer;

[0100] Feature dimension and pooling: The globally enhanced feature matrix H3 is input into the fully connected layer for dimensionality reduction and mean pooling along the sequence dimension:

[0101] h=MeanPool(GELU(W c H3)) (13)

[0102] Among them, W c is the dimensionality reduction matrix, and GELU is the Gaussian error linear unit activation function.

[0103] S502: Output binary logits through the linear transformation layer and apply the cross entropy loss function to train the model.

[0104] The cross entropy loss function is used during training:

[0105]

[0106] Among them, p i =softmax(y i ) is the predicted probability, y i ∈{0,1} is the true label.

[0107] Example 2

[0108] Based on the same inventive concept, this embodiment further provides a short answer scoring system based on the fusion of hierarchical attention and dynamic gating, which is used to implement the method described in Example 1. The system includes:

[0109] Input processing module, used to splice the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector;

[0110] A block attention module is used to process the feature vector through a block context-aware attention mechanism to capture the semantic dependencies within the local block and the semantic interactions across blocks;

[0111] A dynamic gating module, configured to adaptively fuse the local intra-block semantic dependency and cross-block semantic interaction using a dynamic gating mechanism to obtain a fused multi-level semantic representation;

[0112] The global attention module is used to model global long-range semantic associations through stacked multi-head self-attention layers to compensate for semantic omissions in the chunking mechanism;

[0113] The scoring output module is used to calculate the similarity between the reference answer and the student's answer text based on the fused multi-level semantic representation, and output the scoring result.

[0114] The short answer scoring system based on the fusion of hierarchical attention and dynamic gating provided in this embodiment has all the advantages of the short answer scoring method based on the fusion of hierarchical attention and dynamic gating provided in Example 1.

[0115] Example 3

[0116] This embodiment further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0117] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A short answer scoring method based on the fusion of hierarchical attention and dynamic gating, characterized by: The following steps are involved: Concatenate the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector; Processing the feature vectors through a block-wise context-aware attention mechanism to capture local intra-block semantic dependencies and cross-block semantic interactions; A dynamic gating mechanism is used to adaptively fuse the local intra-block semantic dependency and cross-block semantic interaction to obtain a fused multi-level semantic representation; The stacked multi-head self-attention layer models global long-range semantic associations to compensate for semantic omissions in the chunking mechanism. Based on the fused multi-level semantic representation, the similarity between the reference answer and the student's answer text is calculated, and the scoring result is output.

2. The method according to claim 1, characterized in that The formula for concatenating the reference answer text and the student answer text into a structured input sequence is: Input=[CLS]⊕R⊕[SEP]⊕S⊕[SEP]; Among them, [CLS] is the classification mark, R is the reference answer, [SEP] is the separator, S is the student answer, and ⊕ represents sequence splicing.

3. The method according to claim 1, characterized in that The block-wise context-aware attention mechanism includes: Divide the input sequence into K consecutive semantic blocks; Calculate the scaled dot product attention within each semantic block to generate intra-block attention features; A sliding window mechanism is used to make the current block interact with the previous j historical blocks across blocks to generate cross-block attention features.

4. The method according to claim 3, characterized in that The calculation formula for the cross-block attention interaction is: Among them, h is the number of attention heads, is the scaling factor, d is the embedding dimension, Q i is the query vector of the i-th block, is the keyword vector of the first j blocks, V j is the value vector of the first j blocks.

5. The method according to claim 1, characterized in that The dynamic gating mechanism includes: Generate dimension-sensitive gating weight matrix through Sigmoid function; According to the formula Weighted fusion of intra-block attention features and cross-block attention features; where ⊙ represents element-by-element multiplication, A i is the result of the gated fusion of the i-th block, G i is the weight of the gated fusion of the i-th block, is the result of self-attention calculation within the i-th block, is the attention interaction result between the i-th block and the previous i blocks.

6. The method according to claim 5, characterized in that The generation formula of the gating weight matrix is: G=σ(W g ·RMSNorm(H0); Among them, σ is the Sigmoid function, W g is a learnable parameter, RMSNorm is root mean square normalization, and H0 is the context-aware embedding matrix.

7. The method according to claim 1, characterized in that The process of modeling global long-range semantic associations includes: The semantic association between any positions in the entire sequence is calculated through the standard Softmax attention layer; The output is fused with the block attention features through residual connection and RMSNorm layer to generate global enhanced features.

8. The method according to claim 7, characterized in that The steps of calculating the similarity between the reference answer and the student's answer text and outputting the scoring result include: Perform mean pooling on the global enhanced features after dimensionality reduction through a fully connected layer; The binary logits are output through the linear transformation layer, and the cross entropy loss function is applied to train the model.

9. A short answer scoring system based on the fusion of hierarchical attention and dynamic gating, characterized by: For implementing the method according to any one of claims 1 to 8, the system comprises: Input processing module, used to splice the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector; A block attention module is used to process the feature vector through a block context-aware attention mechanism to capture the semantic dependencies within the local block and the semantic interactions across blocks; A dynamic gating module, configured to adaptively fuse the local intra-block semantic dependency and cross-block semantic interaction using a dynamic gating mechanism to obtain a fused multi-level semantic representation; The global attention module is used to model global long-range semantic associations through stacked multi-head self-attention layers to compensate for semantic omissions in the chunking mechanism; The scoring output module is used to calculate the similarity between the reference answer and the student's answer text based on the fused multi-level semantic representation, and output the scoring result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images