Short answer scoring method and system based on hierarchical attention and dynamic gate fusion

Abstract

The application discloses a short answer scoring method and system based on hierarchical attention and dynamic gate fusion, and belongs to the field of natural language processing and intelligent education. The method comprises the following steps: splicing a reference answer text and a student answer text into a structured input sequence, and converting the structured input sequence into a feature vector; processing the feature vector through a block context perception attention mechanism to capture local block semantic dependence and cross-block semantic interaction; adaptively fusing the local block semantic dependence and the cross-block semantic interaction through a dynamic gate mechanism to obtain a fused multi-level semantic representation; modeling global long-range semantic association through a superimposed multi-head self-attention layer to compensate for semantic omission of the block mechanism; and calculating the similarity between the reference answer and the student answer text based on the fused multi-level semantic representation, and outputting a scoring result. Through the three-level attention architecture of "local-cross-block-global", the application solves the imbalance problem between local focusing and global coherence in single-stage fusion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of natural language processing and intelligent education, and particularly relates to a short answer scoring method and system based on hierarchical attention and dynamic gating fusion. BACKGROUND

[0002] In the field of automatic scoring of short answers, traditional methods mainly include rule-based methods, traditional machine learning-based methods and deep learning-based pre-trained models. The rule-based method evaluates the similarity of answers by manually designing matching patterns. This method has high interpretability in specific fields. The traditional machine learning-based model relies on artificially constructed features, including word frequency, TF-IDF, word vector, etc., to capture basic semantic information of text. The deep learning-based pre-trained model automatically learns semantic representation using BERT architecture and models text association through global attention mechanism.

[0003] However, the prior art has significant defects: first, the rule-based method and the traditional machine learning model are difficult to handle non-standard expressions and implicit logical reasoning in student answers, and are not sensitive to local context semantic changes; second, the homogeneous global attention mechanism of the pre-trained model causes irrelevant words to interfere with key semantic units, and the single-layer feature fusion structure cannot balance local fine-grained matching and global coherence, resulting in ineffective modeling of cross-level semantic association, which ultimately restricts the scoring accuracy. SUMMARY

[0004] To solve the above technical problems, the present application provides a short answer scoring method and system based on hierarchical attention and dynamic gating fusion to solve the problems existing in the prior art.

[0005] To achieve the above purpose, in a first aspect, the present application provides a short answer scoring method based on hierarchical attention and dynamic gating fusion, comprising:

[0006] The reference answer text and the student answer text are spliced into a structured input sequence and converted into a feature vector;

[0007] The feature vector is processed by a block context-aware attention mechanism to capture local intra-block semantic dependency and cross-block semantic interaction;

[0008] A dynamic gating mechanism is used to adaptively fuse the local intra-block semantic dependency and the cross-block semantic interaction to obtain a fused multi-level semantic representation;

[0009] A multi-head self-attention layer is used to model global long-range semantic association to compensate for the semantic omission of the block mechanism;

[0010] Based on the fused multi-level semantic representation, the similarity between the reference answer and the student answer text is calculated, and the scoring result is output.

[0011] Preferably, the formula for splicing the reference answer text and the student answer text into a structured input sequence is:

[0012]

[0013] Where [CLS] is a classification mark, R is a reference answer, [SEP] is a separator, S is a student answer, represents sequence splicing.

[0014] Preferably, the block context-aware attention mechanism comprises:

[0015] Divide the input sequence into K continuous semantic blocks;

[0016] Calculate the scaled dot-product attention within each semantic block to generate intra-block attention features;

[0017] Use a sliding window mechanism to make the current block interact with the previous j historical blocks for cross-block attention, to generate cross-block attention features.

[0018] Preferably, the calculation formula of the cross-block attention interaction is:

[0019]

[0020] Where h is the number of attention heads, is a scaling factor, d is an embedding dimension, Q i is the query vector of the i-th block, is the key vector of the previous j blocks, V j is the value vector of the previous j blocks.

[0021] Preferably, the dynamic gating mechanism comprises:

[0022] Generate a dimension-sensitive gating weight matrix through a Sigmoid function;

[0023] According to the formula A i =G i ⊙(A intra +A inter ) weightedly fuses the intra-block attention features and the cross-block attention features; where ⊙ represents element-wise multiplication, A i is the result of the i-th block gating fusion, G i is the weight of the i-th block gating fusion, is the result of the i-th block intra-attention calculation, is the result of the i-th block attention interaction with the previous i blocks.

[0024] Preferably, the generation formula of the gating weight matrix is:

[0025] F = σ(W g ·RMSNorm(H0));

[0026] wherein σ is a Sigmoid function, W g is a learnable parameter, RMSNorm is a root mean square normalization, and H0 is a context-aware embedding matrix.

[0027] Preferably, the process of modeling global long-range semantic association comprises:

[0028] calculating semantic association between any positions in the full sequence through a standard Softmax attention layer;

[0029] fusing the output with the block attention feature through a residual connection and an RMSNorm layer to generate a global enhanced feature.

[0030] Preferably, the step of calculating the similarity between the reference answer and the student answer text and outputting a score result comprises:

[0031] performing mean pooling after reducing the dimension of the global enhanced feature through a fully connected layer;

[0032] outputting binary classification logits through a linear transformation layer and applying a cross-entropy loss function to train the model.

[0033] In a second aspect, the present application also discloses a short answer scoring system based on hierarchical attention and dynamic gating fusion, which is used to implement the method of the first aspect, and the system comprises:

[0034] an input processing module configured to concatenate the reference answer text and the student answer text into a structured input sequence and convert the structured input sequence into a feature vector;

[0035] a block attention module configured to process the feature vector through a block context-aware attention mechanism to capture local intra-block semantic dependency and cross-block semantic interaction;

[0036] a dynamic gating module configured to adaptively fuse the local intra-block semantic dependency and the cross-block semantic interaction through a dynamic gating mechanism to obtain a fused multi-level semantic representation;

[0037] a global attention module configured to model global long-range semantic association through a stacked multi-head self-attention layer to compensate for semantic omission of the block mechanism;

[0038] a scoring output module configured to calculate the similarity between the reference answer and the student answer text based on the fused multi-level semantic representation and output a score result.

[0039] In a third aspect, the present application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in the first aspect.

[0040] Compared with the prior art, the present application has the following advantages and technical effects:

[0041] The present application provides a short answer scoring method based on hierarchical attention and dynamic gating fusion, comprising: firstly, splicing the reference answer text and the student answer text into a structured input sequence and converting it into a feature vector; secondly, processing the feature vector through a block context-aware attention mechanism to capture local intra-block semantic dependency and cross-block semantic interaction; thirdly, adaptively fusing the local intra-block semantic dependency and the cross-block semantic interaction by using a dynamic gating mechanism to obtain a fused multi-level semantic representation; fourthly, modeling global long-range semantic association through a superimposed multi-head self-attention layer to compensate for the semantic omission of the block mechanism; and finally, calculating the similarity between the reference answer and the student answer text based on the fused multi-level semantic representation, and outputting a scoring result.

[0042] The present application explicitly divides semantic units through a block context-aware attention mechanism, effectively analyzes non-standard expressions and implicit reasoning logic in student answers under the dual action of intra-block self-attention and cross-block sliding window interaction, and improves the sensitivity to local subtle semantic changes.

[0043] The present application adaptively adjusts the fusion weight of local block features and cross-block interaction features based on a dynamic gating mechanism, suppresses noise interference and enhances the discriminability of key semantic units, filters irrelevant word interference, and strengthens the representation ability of discriminative semantic units.

[0044] The present application realizes balanced modeling of cross-level semantics through a "local-cross-block-global" three-level attention architecture, focuses on fine-grained matching through a block mechanism, realizes global Softmax attention compensation for long-range dependence, and solves the imbalance problem between local focus and global coherence in single-stage fusion.

[0045] The present application significantly improves the analysis accuracy of cross-level semantic association in fragmented text by performing similarity calculation based on the optimized multi-level semantic representation, ensures the scoring reliability in complex contexts, and improves the scoring accuracy and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and its description are used to explain the application. Do not constitute an improper limitation on the application. In the drawings:

[0047] Figure 1 The flowchart of the short answer scoring method based on hierarchical attention and dynamic gating fusion of the embodiments of the present application;

[0048] Figure 2 Figure 1 is a cross-block interaction diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0049] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0051] Embodiment one

[0052] As shown in Figure 1 , the present embodiment provides a short answer scoring method based on hierarchical attention and dynamic gating fusion, comprising:

[0053] S1, splice the reference answer text and the student answer text into a structured input sequence, and convert it into a feature vector;

[0054] Input sequence splicing: splice the reference answer (R) and the student answer (S) into a structured input sequence in the following format:

[0055]

[0056] Wherein, [CLS] is a classification mark, [SEP] is a separator, represents sequence splicing. For example, if the reference answer is "photosynthesis needs light", and the student answer is "plants carry out photosynthesis in sunlight", then the spliced input is:

[0057] [CLS] photosynthesis needs light [SEP] plants carry out photosynthesis in sunlight [SEP].

[0058] S2, process the feature vector through a block context-aware attention mechanism to capture local block-in semantic dependency and cross-block semantic interaction;

[0059] Context embedding generation: encode the input sequence using a pre-trained DeBERTa-v3 model (embedding dimension d = 768) to output an embedding matrix with context awareness:

[0060]

[0061] Wherein, L is the length of the input sequence (maximum support 512 tokens), and d = 768 is the embedding dimension.

[0062] Capturing local and cross-block semantic dependencies through a chunked context-aware attention mechanism, specifically including:

[0063] Blocking strategy: Divide the input sequence into blocks. Each semantic block has a block length B = 8, and the sliding window step size is the same as the block length. If there are fewer than B tokens at the end, they are padded with zeros. For example, when L = 100, it is divided into 13 blocks (12 complete blocks + 1 padded block).

[0064] Projection calculation: for each block Perform linear projection and activation:

[0065] Q i ,K i V i =SiLU(W q H i ),SiLU(W k H i ),SiLU(W v H i (3)

[0066] Among them, Q i Let K be the query vector for the i-th block. i V is the key vector of the i-th block. i Let i be the value vector of the i-th block, where the right subscript i represents the i-th block. Q i ,K i V i Learnable parameters; the SiLU activation function is defined as SiLU(x)=x·σ(x), which is the Sigmoid linear activation function; Let B be the input of the i-th block, B be the block length, and d be the embedding dimension.

[0067] Intra-block cross-block attention calculation:

[0068] Intra-block attention: Calculates the attention relationships between tokens within the current block.

[0069]

[0070] Where h = 8 is the number of attention heads. d is the scaling factor, d is the embedding dimension, and Q is the embedding dimension. i Let i be the query vector for the i-th block. V is the keyword vector of the first i blocks. i Let i be the value vector of the first i blocks.

[0071] Inter-block: To enhance the continuity of local semantics and avoid information fragmentation, a sliding window mechanism is adopted, and the sliding window step is set to be equal to B. If the sequence is not long enough for one block, it is padded with zeros to B tokens. Figure 2 The inter-block interaction diagram is shown; wherein Figure 2 (a) in (a) is to calculate the global interaction graph of the first two historical blocks and the current second block, Figure 2 (b) in (b) is to calculate the global interaction graph of the first three historical blocks and the current third block, Figure 2 (c) in (c) is to calculate the global interaction graph of the first n historical blocks and the current n block.

[0072] The global interaction formula for calculating the first j historical blocks and the current i block is:

[0073]

[0074] where j<=i, i>1, h=8 is the number of attention heads, is a scaling factor, d is the embedding dimension, Q i is the query vector of the i-th sub-block, is the key vector of the first j sub-blocks, V j is the value vector of the first j sub-blocks.

[0075] The embodiment captures the continuity of local semantics and avoids information fragmentation through inter-block interaction.

[0076] S3, adopt dynamic gating mechanism to adaptively fuse the local block semantic dependence and cross-block semantic interaction, and obtain the fused multi-level semantic representation;

[0077] Further, the local and cross-block attention is fused, and the output is modulated by a gating weight;

[0078] Dynamic gating fusion: generate a gating weight matrix G∈[0,1] L×d , to adaptively adjust the feature fusion strength:

[0079] G=σ(W g ·RMSNorm(H0))∈[0,1] L×d (6)

[0080] where, σ is a Sigmoid function, W g is a learnable parameter, RMSNorm is a root mean square normalization, and H0 is a context-aware embedding matrix.

[0081] Finally, the block attention output is weighted by the gating:

[0082]

[0083] where denotes element-wise multiplication, A i is the result of the i-th block gated fusion, G i is the weight of the i-th block gated fusion, is the result of the i-th block intra-attention computation, is the result of the i-th block attention interaction with the previous i-1 blocks.

[0084] In this embodiment, to alleviate the gradient vanishing problem in deep network training, residual connection and RMSNorm are introduced. After concatenating the block attention outputs, residual connection and RMSNorm are used to stabilize the training:

[0085] H1=W o ·Concat(A1,…,A K ) (8)

[0086] H2=RMSNorm(H0+H1) (9)

[0087] where W o is the output projection matrix.

[0088] S4, modeling global long-range semantic association through stacked multi-head self-attention layers to compensate for the semantic omission of the block mechanism;

[0089] Specifically, the long-range semantic association between any positions in the full sequence is established through stacked multi-head self-attention layers, which specifically includes:

[0090] To supplement the long-range dependencies that may be omitted by the block mechanism, a standard multi-head Softmax attention layer is stacked to repair the global association that may be omitted by the block mechanism:

[0091] Q'=W q 'H2, K'=W k 'H2, V'=W v 'H2 (10)

[0092]

[0093] where d h =d / h=96, the number of attention heads h=8.

[0094] The output is connected through residual connection and normalization:

[0095] H3=RMSNorm(H2+A') (12)

[0096] This layer explicitly models the semantic association between any positions in the full sequence, which is complementary to the block attention.

[0097] S5, based on the fused multi-level semantic representation, calculating the similarity between the reference answer and the student answer text, and outputting the scoring result.

[0098] Further, the step of calculating the similarity between the reference answer and the student answer text and outputting the scoring result comprises:

[0099] S501, performing mean pooling after reducing the dimension of the global enhanced features through a fully connected layer;

[0100] Feature dimension and pooling: input the global enhanced feature matrix H3 into a fully connected layer for dimension reduction, and perform mean pooling along the sequence dimension:

[0101] h=MeanPool(GELU(W c H3)) (13)

[0102] Wherein, W c is the dimension reduction matrix, and GELU is the Gaussian error linear unit activation function.

[0103] S502, outputting binary classification logits through a linear transformation layer, and applying a cross-entropy loss function to train the model.

[0104] Cross-entropy loss function is used during training:

[0105]

[0106] Wherein, p i =softmax(y i ) is the predicted probability, and y i ∈{0,1} is the true label.

[0107] Embodiment two

[0108] Based on the same inventive concept, the embodiment also provides a short answer scoring system based on hierarchical attention and dynamic gating fusion, which is used to realize the method of embodiment one. The system comprises:

[0109] An input processing module is used to splice the reference answer text and the student answer text into a structured input sequence and convert it into a feature vector;

[0110] A block attention module is used to process the feature vector through a block context-aware attention mechanism to capture local intra-block semantic dependencies and cross-block semantic interactions;

[0111] A dynamic gating module is used to adaptively fuse the local intra-block semantic dependencies and cross-block semantic interactions through a dynamic gating mechanism to obtain a fused multi-level semantic representation;

[0112] A global attention module is configured to model global long-range semantic associations through stacked multi-head self-attention layers to compensate for semantic omissions of the block mechanism.

[0113] A score output module is configured to calculate the similarity between the reference answer and the student answer text based on the fused multi-level semantic representation, and output a score result.

[0114] The short answer scoring system based on hierarchical attention and dynamic gating fusion has all the advantages of the short answer scoring method based on hierarchical attention and dynamic gating fusion of the first embodiment.

[0115] Embodiment three

[0116] The embodiment also discloses a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the method in the first embodiment.

[0117] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A short answer scoring method based on hierarchical attention and dynamic gate fusion, characterized in that, The method comprises the following steps: splicing the reference answer text and the student answer text into a structured input sequence and converting the structured input sequence into a feature vector; processing the feature vector through a block context-aware attention mechanism to capture local intra-block semantic dependencies and cross-block semantic interactions; adopting a dynamic gating mechanism to adaptively fuse the local intra-block semantic dependencies and the cross-block semantic interactions to obtain a fused multi-level semantic representation; modeling global long-range semantic associations through a stacked multi-head self-attention layer to compensate for semantic omissions of the block mechanism; calculating the similarity between the reference answer and the student answer text based on the fused multi-level semantic representation and outputting a scoring result.

2. The method of claim 1, wherein a formula for splicing the reference answer text and the student answer text into a structured input sequence is: Input = [CLS] O R O [SEP] O S O [SEP]; wherein [CLS] is a classification mark, R is a reference answer, [SEP] is a separator, S is a student answer, and O represents sequence splicing.

3. The method of claim 1, wherein the block context-aware attention mechanism comprises: dividing the input sequence into K continuous semantic blocks; calculating scaled dot-product attention within each semantic block to generate intra-block attention features; adopting a sliding window mechanism to enable cross-block attention interaction between the current block and the previous j historical blocks to generate cross-block attention features.

4. The method of claim 3, wherein a calculation formula for the cross-block attention interaction is:

5. The method of claim 1, wherein the dynamic gating mechanism comprises: generating a dimension-sensitive gating weight matrix through a Sigmoid function; 6. The method of claim 5, wherein a generation formula for the gating weight matrix is: where h is the number of attention heads, is a scaling factor, d is the embedding dimension, Q i is the query vector of the i-th block, is the key vector of the first j blocks, V j is the value vector of the first j blocks.

7. The method of claim 1, wherein the process of modeling global long-range semantic associations comprises: calculating semantic associations between any positions in the entire sequence through a standard Softmax attention layer; fusing the output and the block attention features through a residual connection and an RMSNorm layer to generate global enhanced features. According to the formula The weighted fusion block attention feature and the cross-block attention feature; wherein, ⊙ represents element-by-element multiplication, A i G is the result of the i-th sub-block gating fusion, i G is the weight of the i-th sub-block gating fusion, G is the i-th sub-block intra-block self-attention calculation result, G is the i-th sub-block and the previous i sub-block attention interaction result.

8. The method of claim 7, wherein the step of calculating the similarity between the reference answer and the student answer text and outputting a scoring result comprises: performing mean pooling after reducing the dimension of the global enhanced features through a fully connected layer; G = σ(W g RMSNorm(H0)); where σ is a sigmoid function, W g is a learnable parameter, RMSNorm is a root mean square normalization, and H0is a context-aware embedding matrix. outputting binary classification logits through a linear transformation layer and applying a cross-entropy loss function to train the model. A system for implementing the method of any one of claims 1-8 comprises: an input processing module for splicing the reference answer text and the student answer text into a structured input sequence and converting the structured input sequence into a feature vector; a block attention module for processing the feature vector through a block context-aware attention mechanism to capture local intra-block semantic dependencies and cross-block semantic interactions; a dynamic gating module for adopting a dynamic gating mechanism to adaptively fuse the local intra-block semantic dependencies and the cross-block semantic interactions to obtain a fused multi-level semantic representation; ​ ​ ​ 9. A short answer scoring system based on hierarchical attention and dynamic gate fusion, characterized in that, ​ ​ ​ ​ A global attention module is configured to model global long-range semantic associations through stacked multi-head self-attention layers to compensate for semantic omissions of the chunking mechanism. A score output module is configured to calculate the similarity between the reference answer and the student answer text based on the fused multi-level semantic representation, and output a score result.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1-8.

Images