Cross-scene English automatic composition scoring system and method based on known prompt

Abstract

The invention discloses a cross-scene English automatic composition scoring system and method based on known prompts, and the system comprises a composition superficial layer feature extraction module which is used for extracting superficial layer language features of a composition, i.e., part-of-speech features, and is used for coding the composition needing to be scored, and obtaining superficial layer feature representation of the composition; the prompt shallow layer feature extraction module is used for extracting prompted shallow layer language features including grammar features and syntactic features, and is used for encoding prompts to which the to-be-scored composition belongs to obtain prompted shallow layer feature representation; the prompt-composition correlation extraction module is used for extracting the relation between the prompt and the composition and judging whether the composition meets the requirements of the theme or not; and the feature fusion training prediction module is used for carrying out feature connection operation on the extracted features, composition codes and prompt codes, and is used for final composition scoring. According to the method, the accuracy of the model is improved, and the scoring result has higher consistency.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and smart education technology, and in particular to a system and method for automatically grading English compositions across scenarios based on known prompts. Background Art

[0002] Automatic essay scoring refers to the process of automatically scoring students' or examinees' essays using natural language processing technology and artificial intelligence technologies such as machine learning and deep learning. It aims to automatically and accurately evaluate the quality of a given essay and provide fast, objective and consistent scoring results.

[0003] Existing automatic essay scoring technologies mainly focus on two categories of scenarios: automatic essay scoring for specific prompts and automatic essay scoring across prompt scenarios. Automatic essay scoring for specific prompts means that the unscored essays and the scored essays come from the same essay prompt, that is, the training set and test set used in the study belong to the same prompt and follow the same experimental settings. In the early stages, such research relied on manual feature extraction methods, focusing mainly on surface features such as vocabulary and sentence structure. Although satisfactory results were achieved, this method requires a lot of manpower. With the rise of deep learning, many automatic essay scoring systems based on neural network models have been proposed, which improves the accuracy of scoring.

[0004] However, in real-world educational scenarios, there are no pre-labeled essays for specific prompts, so the actual application effect of automatic essay scoring for specific prompts is not good. Therefore, automatic essay scoring across prompt scenarios has become the mainstream in recent years. Its advantage is that it is more in line with the scoring tasks of real-world educational scenarios and is easier to handle the scoring tasks of essays of different themes. At the same time, this task also puts higher requirements on the scoring system's ability to extract article feature representations and generalization capabilities. Summary of the invention

[0005] The purpose of the present invention is to provide a cross-scenario English automatic composition scoring system and method based on known prompts to address the problems of poor generalization ability, poor interpretability and large differences from actual scoring scenarios in previous automatic composition scoring technologies, which improves the accuracy and consistency of scoring, reduces the pressure of manual scoring and reduces the cost of composition evaluation.

[0006] The technical solution adopted by the present invention is:

[0007] A method for automatic English composition scoring based on known prompts across scenarios, comprising the following steps:

[0008] Step 1: Extract shallow language features of the composition: adopt the feature representation method of part-of-speech mapping, and obtain the final shallow feature representation of the composition through a series of neural network layers;

[0009] Step 2: Extract the shallow language features of the prompt: Use a feature representation method that combines part-of-speech tagging and syntactic features, and obtain the final shallow feature representation of the prompt through a series of neural network layers

[0010] Step 3: Extract the relationship between the prompt and the composition; at the same time, compare the semantics of the prompt with the entire article and the fragment to extract the prompt-composition correlation features to judge whether the composition meets the requirements of the theme;

[0011] Step 4: Perform feature connection operation on the extracted features, composition code, and prompt code to obtain the final composition score.

[0012] Furthermore, step 1 specifically includes the following steps:

[0013] Step 1-1, performing word-level encoding representation of the composition, specifically includes the following steps:

[0014] Step 1-1-1, segment the sentence and then perform data cleaning;

[0015] Step 1-1-2, use the NLTK package in Python to perform part-of-speech tagging on each word, generate a part-of-speech dictionary through statistical methods, and map each word to the corresponding vector space. The word vector sequence after composition mapping is (P1, P2, ..., P i ,…,P n ), P i is the dense vector of the i-th word in the composition, and n is the total number of words;

[0016] Step 1-1-3, use the word-level Dropout layer to randomly discard a certain percentage of neuron outputs to obtain the feature representation P′ i , reducing the model's dependence on specific neurons and improving the model's generalization ability. The specific expression is as follows:

[0017] P′ i = Dropout(P i ) (1)

[0018] Among them, P′ i It is the feature representation after the Dropout layer;

[0019] Step 1-2, perform sentence-level encoding of the composition: specifically includes the following steps:

[0020] Step 1-2-1, represent the feature P′ i Learn low-level feature representations useful for the scoring task through a shared 1D convolutional encoder with an attention layer;

[0021] The expression of the shared 1D convolutional encoder is as follows:

[0022] c i =f(W c ·[P′ i :P′ i+l-1 ]+b c ) (2)

[0023] The expression of the attention mechanism processing method is as follows:

[0024] a i =tanh(W a ·c i +b a ) (3)

[0025]

[0026] Among them, W c is the weight matrix, b c is the bias vector, l is the window size of the CNN layer, c i is the result of convolution calculation; a i is the attention vector of the i-th sentence, W a and b a is used to generate the attention vector a i The weight matrix and bias vector of u is the weight vector, a j Represents the attention vector of a sentence in the composition (also including a i ), the maximum value of j is the number of sentences in the composition, u i is the attention weight; s t represents the final feature representation of the tth sentence after the pooling operation of the attention layer, and S represents the set of all sentence representations in the composition obtained after the pooling operation of the attention layer, that is, S = [s1, s2, ..., s m ],s m Indicates the last sentence of a composition;

[0027] Step 1-2-2, add a multi-head attention mechanism to capture the relationship between different features of the composition and improve the accuracy of the score. The corresponding calculation formula is as follows:

[0028]

[0029]

[0030] Among them, Attention represents the scaling dot product attention mechanism, represents the i-th attention mechanism head, h represents the number of attention heads, represents the last attention mechanism head, is the parameter matrix corresponding to query, key, and value in the multi-head attention mechanism; j represents the feature task for scoring the essay (for example, the feature tasks include: overall score, content, word choice, sentence fluency, etc.); Concat means connecting the results of all attention mechanism heads, W jo is the dot product weight matrix;

[0031] This task grades an essay, not just the final overall score, but also the content, word choice, sentence fluency, etc. This can improve the interpretability of the essay, such as what makes it good.

[0032] Step 1-2-3, assume that the sentence representation set after the multi-head attention mechanism is H = mhead(S) j ,[h1,h2,…,h m ] represent the representation of each sentence in the composition, and then perform sentence-level Dropout layer operation to prevent overfitting of the model;

[0033] h′ i = Dropout(h i ) (8)

[0034] Among them, h′ i is the i-th sentence h i Feature representation after the Dropout layer;

[0035] Steps 1-3, perform article-level encoding representation of the composition: use the long short-term memory network layer LSTM with an attention layer to perform article-level feature representation; the expression of LSTM is as follows:

[0036]

[0037] The input is a sentence representation [h′1,h′2,…h′ m ], represents the hidden representation of the jth feature scoring task after LSTM with a step length of t; LSTM will automatically traverse all time steps t of this sequence, that is, the value range of step length t is 1-m; h′ t-1 It represents the sentence representation when the step length is t-1, h′ t It represents the sentence representation with the current step length of t. The two are used as input to get

[0038] Furthermore, the number of attention heads is set to 2 in step 1-2-2.

[0039] Furthermore, step 2 specifically includes the following steps:

[0040] Step 2-1, performing word-level encoding representation of the prompt; specifically comprising the following steps:

[0041] Step 2-1-1, segment the sentence and then perform data cleaning;

[0042] Step 2-1-2, use the NLTK package in Python to perform part-of-speech tagging on each word, generate a part-of-speech dictionary through statistical methods, and map each word to the corresponding vector space. The sequence of part-of-speech features after the mapping is (tops1, tops2, …, tops i ,…,tops n ); At the same time, the Spacy package in Python is used to extract the syntactic features of each word, and a syntactic dictionary is generated through statistical methods, and each word is mapped to the corresponding vector space. The syntactic feature sequence after the mapping is (tsyn1, tsyn2, …, tsyn i ,…,tsyn n );

[0043] Step 2-1-3, add the part-of-speech feature sequence and syntactic feature sequence after prompt mapping element by element to obtain the mapping Q of each word in the prompt in the vector space i , the calculation formula is Q i =Add(tpos i ,tsyn i );

[0044] In step 2-1-4, a certain percentage of neuron outputs are randomly discarded using a word-level Dropout layer to reduce the model’s reliance on specific neurons, thereby improving the model’s generalization ability. The formula is as follows (1-1):

[0045] Q′ i = Dropout(Q i ) (1-1)

[0046] Among them, Q i The dense vector representing the i-th word of the prompt, Q′ i It is the feature representation after the Dropout layer;

[0047] Step 2-2, performing sentence-level encoding representation of the prompt; specifically comprising the following steps:

[0048] Step 2-2-1, Q′ i Feed into a shared 1D convolutional encoder with an attention layer to learn low-level feature representations useful for the scoring task;

[0049] The expression of the shared 1D convolutional encoder is as follows:

[0050] c′ i =f(W′ c ·[Q′ i :Q′ i+l-1 ]+b′ c ) (2-1)

[0051] The expression of the attention mechanism processing method is as follows:

[0052] a′ i =tanh(W′ a ·cv i +b′ a ) (3-1)

[0053]

[0054] Among them, W′ c is the weight matrix, b′ c is the bias vector, l′ is the window size of the CNN layer, c′ i is the result of convolution calculation; a′ i is the attention vector of the ith sentence, W′ a and b′ a is used to generate the attention vector a′ i The weight matrix and bias vector of u is the weight vector, a′ j Represents the attention vector of a sentence in the prompt (also contains a′ i ), the maximum value of j is the number of prompt sentences, u′ i is the attention weight; s′ t represents the final feature representation of the tth sentence after the pooling operation of the attention layer, and S represents the set of all sentence representations in the prompt obtained after the pooling operation of the attention layer, that is, S′=[s′1,s′2,…,s′ m ],s′ m Indicates the last sentence of the prompt;

[0055] Step 2-2-2, add a multi-head attention mechanism to capture the relationship between different features of the prompt and improve the accuracy of the score. The corresponding calculation formula is as follows:

[0056]

[0057] Among them, Attention represents the scaling dot product attention mechanism, represents the i-th attention mechanism head, h represents the number of attention heads, represents the last attention mechanism head, is the parameter matrix corresponding to query, key, and value in the multi-head attention mechanism; j represents the feature task to be scored (for example, feature tasks include: overall score, content, word choice, sentence fluency, etc.); Concat means connecting the results of all attention mechanism heads, V jo is the dot product weight matrix;

[0058] Step 2-2-3, assume that the sentence representation set after the multi-head attention mechanism is H2 = mhead(S′) j ,[h21,h22,…,h2 m ] represent the representation of each sentence in the prompt, and then perform sentence-level Dropout layer operation to prevent overfitting of the model;

[0059] h2′ i =Dropout(h2 i ) (8-1)

[0060] Among them, h2′ i is the i-th sentence h2 i Feature representation after the Dropout layer;

[0061] Step 2-3, perform article-level encoding representation of prompts: use the long short-term memory network layer LSTM with an attention layer to perform article-level feature representation; the expression of LSTM is as follows:

[0062]

[0063] The input is the sentence representation [h2′1,h2′2,…h2′ m ], represents the hidden representation of the jth feature scoring task after LSTM with a step length of t; LSTM will automatically traverse all time steps t of this sequence, that is, the value range of step length t is 1-m; h2′ t-1 It represents the sentence representation when the step length is t-1, h2′ t It represents the sentence representation with the current step length of t. The two are used as input to get

[0064] Furthermore, in step 3, the article is first segmented according to the meaning to determine whether the author includes irrelevant content in the article.

[0065] Furthermore, step 3 specifically includes the following steps:

[0066] Step 3-1, without considering the writing prompt, use the existing mature large language model to perform semantic text segmentation on the composition, and divide the input composition into 1-n segments;

[0067] Furthermore, in step 3-1, the Qwen-max large language model launched by Alibaba is used to perform semantic text segmentation on the composition.

[0068] Step 3-2, using cosine similarity to compare the correlation between the prompt and the entire composition, and between the prompt and the composition fragment, to calculate the prompt-composition correlation. The specific calculation formula is as follows:

[0069]

[0070] sim e =cos(essay,prompt) (12)

[0071] sim i =cos(segment i ,prompt) (13)

[0072] Among them, sim e is the cosine similarity between the entire article and the prompt, sim i is the cosine similarity between the current article segment and the prompt, PR represents the final relevance between the prompt and the article; α represents the reward weight of the relevance between the article and the prompt, which is used to strengthen the influence of the number of article segments on the judgment of prompt relevance; γ is a random number between -0.03 and +0.03, which is introduced to prevent overfitting of the training data. The more segments the essay is divided into, the higher the weight. As the number of segments increases, it eventually approaches the limit value of 1 to reduce the excessive influence of the segments on the article.

[0073] Step 3-3, considering that the characteristics of the composition itself cannot reflect whether the composition meets the prompt requirements, the present invention connects the characteristics of the composition itself (such as: simple features such as the number of words and the number of typos) with the extracted prompt-composition correlation features to form the final feature set fs.

[0074] Furthermore, step 4 uses the prompt-essay attention mechanism to effectively capture the relationship between the prompt and the essay, providing effective support for scoring; the specific steps are as follows:

[0075] Step 4-1, use the multi-head attention mechanism to implement the article prompt attention mechanism, where query is the encoded representation of the prompt, and key and value are the encoded representations of the essay;

[0076] This setting allows viewing sub-parts of the prompt at each position in the article. Therefore, article-prompt attention can effectively capture the relationship between prompts and articles, providing effective support for scoring.

[0077] Step 4-2, then connect an LSTM layer, and use the average pooling layer and the maximum pooling layer to obtain the prompt article representation; the formula for the pooling operation is as follows:

[0078]

[0079] Among them, x j represents the output of the LSTM layer, and They represent the results after performing the maximum pooling layer operation and the average pooling layer operation, respectively. j yes and The output obtained after the Concat operation represents the final representation of the j-th scoring task;

[0080] Max pooling and average pooling layers are used here because their advantage over attention pooling layers is that they have fewer parameters while performing well.

[0081] Step 4-3: Get the final representation E of the j-th scoring task j Connect with the extracted final feature set fs: I j =[E j ; fs], forming the final input of the model;

[0082] Step 4-4, then use trait-attention to focus on the representation of other features of the composition. The formula is as follows:

[0083] A=[I 1 ,I 2 ,…,I j ,…,I M ] (17)

[0084]

[0085]

[0086] r j =[I j ; g j ] (20)

[0087] Among them, A represents the connection of each task, A -j represents the masking of the target composition feature representation, is the attention weight of the ith feature, g jand r j Represents the attention vector and the final feature representation;

[0088] Step 4-5, the final score is obtained by the sigmoid function after the linear layer:

[0089] Furthermore, the present invention also discloses a cross-scenario English automatic composition scoring system based on known prompts, which adopts the cross-scenario English automatic composition scoring method based on known prompts, and the system includes:

[0090] Composition shallow feature extraction module: extracts the shallow language features of the composition, namely the part-of-speech features; it is used to encode the composition to be scored and obtain the shallow feature representation of the composition;

[0091] Prompt shallow feature extraction module: extracts the shallow language features of prompts, including grammatical features and syntactic features; used to encode the prompts to which the essay to be graded belongs, and obtains the shallow feature representation of the prompts;

[0092] Prompt-essay correlation extraction module: extracts the relationship between prompts and essays to judge whether the essays meet the requirements of the theme;

[0093] Feature fusion training prediction module: used to perform feature connection operations on the extracted features, composition codes, and prompt codes for the final composition scoring.

[0094] The present invention adopts the above technical solution and has the following technical advantages: (1) The present invention improves the accuracy and consistency of AI scoring, which can reduce the pressure of manual scoring and reduce the cost of essay evaluation. (2) The present invention introduces a new method that combines part-of-speech tagging and syntactic features for the coding of prompts, which can more effectively obtain the shallow feature representation of prompts and improve the accuracy of scoring. (3) The present invention designs a new method to calculate the correlation between prompts and articles. The algorithm uses a large language model to perform text semantic segmentation to obtain short article fragments, and combines cosine similarity to obtain the similarity between prompts and short articles and between short article fragments. Thereby, it is possible to better judge whether the composition meets the requirements of the prompt, thereby improving the accuracy of system scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments;

[0096] Figure 1 The present invention is a schematic diagram of the structure of an automatic English composition scoring system based on known prompts across scenarios. Implementation

[0098] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0099] Through research and analysis of previous automatic essay scoring systems across prompt scenarios, we found that there have been many studies on extracting essay features for scoring, but there are still few scoring tasks that combine the relationship between essays and prompts. However, whether the essay meets the requirements of the prompt is an important criterion for judging the score of the essay. Therefore, Figure 1 As shown, the present invention discloses a cross-scenario English automatic composition scoring system based on known prompts, which includes:

[0100] The composition shallow feature extraction module extracts the shallow language features of the composition, namely, the part-of-speech features, which are used to encode the composition to be scored and obtain the shallow feature representation of the composition;

[0101] A prompt shallow feature extraction module is used to extract the shallow language features of the prompt, including grammatical features and syntactic features, which are used to encode the prompt to which the composition to be graded belongs and obtain the shallow feature representation of the prompt;

[0102] The prompt-composition correlation extraction module extracts the relationship between the prompt and the composition to judge whether the composition meets the requirements of the theme;

[0103] The feature fusion training prediction module is used to perform feature connection operations on the extracted features and composition codes and prompt codes for the final composition scoring.

[0104] The present invention discloses a method for automatic English composition scoring based on known cross-scenario prompts, and the specific steps are as follows:

[0105] Step 1: Extract shallow language features of the composition: adopt the feature representation method of part-of-speech mapping, and obtain the final shallow feature representation of the composition through a series of neural network layers;

[0106] Step 2: Extract the shallow language features of the prompt: Use a feature representation method that combines part-of-speech tagging and syntactic features, and obtain the final shallow feature representation of the prompt through a series of neural network layers

[0107] Step 3: Extract the relationship between the prompt and the composition; at the same time, compare the semantics of the prompt with the entire article and the fragment to extract the prompt-composition correlation features to judge whether the composition meets the requirements of the theme;

[0108] Step 4: Perform feature connection operation on the extracted features, composition code, and prompt code to obtain the final composition score.

[0109] In a specific embodiment, the shallow feature extraction method of the composition in step 1 adopts a feature representation method of part-of-speech mapping, and obtains the final shallow feature representation of the composition through a series of neural network layers.

[0110] Step 1-1: Word-level encoding representation of the composition; specifically includes the following steps:

[0111] Step 1-1-1: Segment the sentence and then perform data cleaning (cleaning special characters);

[0112] Step 1-1-2: Use the NLTK package in Python to tag each word with a part-of-speech tag, generate a POS (part-of-speech) dictionary through statistical methods, and map each word to the corresponding vector space. Assume that the word sequence (w1, w2, ..., w n ) The word vector sequence after mapping is (P1, P2, ..., P n );

[0113] Step 1-1-3: Use the word-level Dropout layer to randomly discard a certain percentage of neuron outputs to reduce the model's dependence on specific neurons, thereby improving the model's generalization ability. The formula is shown below (1):

[0114] P′ i = Dropout(P i ) (1)

[0115] Where P i Represents the i-th word w i The dense vector, P′ i It is the feature representation after the Dropout layer.

[0116] Step 1-2: Sentence-level encoding representation of the composition; specifically includes the following steps:

[0117] Step 1-2-1: P′ i It is fed into a shared 1D convolutional encoder with an attention layer, with the goal of learning low-level feature representations useful for scoring tasks. The shared 1D convolutional encoder is shown in the following formula (2), and formula (3-5) represents the processing method of the attention mechanism:

[0118] c i =f(W c ·[P′ i :P′ i+l-1 ]+b c ) (2)

[0119] a i =tanh(W a ·c i +b a ) (3)

[0120]

[0121] Among them, W c is the weight matrix, b c is the bias vector, l is the window size of the CNN layer, c i is the result of convolution calculation. i is the attention vector of the i-th sentence, W a and b a is used to generate the attention vector a i The weight matrix and bias vector of w u is the weight vector, a j Represents the attention vector of a sentence in the composition (also including a i ), the maximum value of j is the number of sentences in the composition, u i is the attention weight. t represents the final feature representation of the tth sentence after the pooling operation of the attention layer, and S represents the set of all sentence representations in the composition obtained after the pooling operation of the attention layer, that is, S = [s1, s2, ..., s m ],s m Indicates the last sentence of the composition.

[0122] Step 1-2-2: Adding a multi-head attention mechanism to capture the relationship between different features of the essay can effectively improve the accuracy of the score. Here, the number of attention heads is set to 2; the following formulas (6) and (7) show the calculation formula:

[0123]

[0124] Among them, Attention represents the scaling dot product attention mechanism, represents the i-th attention mechanism head, h represents the number of attention heads, represents the last attention mechanism head, is the parameter matrix corresponding to query, key, and value in the multi-head attention mechanism. j represents the feature task for scoring the essay (for example, feature tasks include: overall score, content, word choice, sentence fluency, etc.); Concat means connecting the results of all attention mechanism heads, W jo is the dot product weight matrix.

[0125] Step 1-2-3: Assume that the sentence representation set after the multi-head attention mechanism is H = mhead(S) j ,[h1,h2,…,h m] represent the representation of each sentence in the composition, and then perform sentence-level Dropout layer operation to prevent overfitting of the model;

[0126] h′ i = Dropout(h i ) (8)

[0127] Among them, h′ i is the i-th sentence h i Feature representation after the Dropout layer;

[0128] Step 1-3: Article-level encoding representation of the composition: Use a long short-term memory network layer (LSTM) with an attention layer to perform article-level feature representation; the attention layer is shown in formulas (3), (4) and (5) above, and the LSTM is shown in formula (9):

[0129]

[0130] The input is a sentence representation [h′1,h′2,…h′ m ], represents the hidden representation of the jth feature scoring task after LSTM with a step length of t; LSTM will automatically traverse all time steps t of this sequence, that is, the value range of step length t is 1-m; h′ t-1 It represents the sentence representation when the step length is t-1, h′ t It represents the sentence representation with the current step length of t. The two are used as input to get The application of the LSTM layer enables the model to capture the sequential dependencies and long-range dependencies in the article, so that the model can learn contextual representations and extract useful features; the attention mechanism layer allows the model to pay more attention to the important information in the article. Finally, the final shallow feature representation of the composition is output.

[0131] In a specific embodiment, the prompt shallow feature extraction method adopts a feature representation method combining part-of-speech tagging and syntactic features, and obtains the final shallow feature representation of the prompt through a series of neural network layers. The specific steps are as follows:

[0132] Furthermore, step 2 specifically includes the following steps:

[0133] Step 2-1, performing word-level encoding representation of the prompt; specifically comprising the following steps:

[0134] Step 2-1-1, segment the sentence, and then perform data cleaning (cleaning special characters);

[0135] Step 2-1-2, use the NLTK package in Python to tag each word, generate a part-of-speech dictionary POS through statistical methods, and map each word to the corresponding vector space. Assume that the word sequence (w21, w22, ..., w2 n ) After mapping, we get the prompt that the sequence of part-of-speech features after mapping is (tops1, tops2, …, tops i ,…,tops n ); At the same time, the Spacy package in Python is used to extract the syntactic features of each word, and a syntactic dictionary Syn is generated through statistical methods, and each word is mapped to the corresponding vector space. Assume that the word sequence (w21, w22, …, w2 n ) After mapping, we get the following syntactic feature sequence: (tsyn1, tsyn2, …, tsyn i ,…,tsyn n );

[0136] Step 2-1-3, perform the Add operation on the part-of-speech feature sequence and the syntactic feature sequence after the prompt mapping, that is, add the two tensors element by element to obtain the mapping Q of each word in the prompt in the vector space i , the calculation formula is Q i =Add(tpos i ,tsyn i );

[0137] In step 2-1-4, a certain percentage of neuron outputs are randomly discarded using a word-level Dropout layer to reduce the model’s reliance on specific neurons, thereby improving the model’s generalization ability. The formula is as follows (1-1):

[0138] Q′ i = Dropout(Q i ) (1-1)

[0139] Among them, Q i The dense vector representing the i-th word of the prompt, Q′ i It is the feature representation after the Dropout layer;

[0140] Step 2-2, performing sentence-level encoding representation of the prompt; specifically comprising the following steps:

[0141] Step 2-2-1, Q′ i Feed into a shared 1D convolutional encoder with an attention layer to learn low-level feature representations useful for the scoring task;

[0142] The expression of the shared 1D convolutional encoder is as follows:

[0143] c′i =f(W′ c ·[Q′ i :Q′ i+l-1 ]+b′ c ) (2-1)

[0144] The expression of the attention mechanism processing method is as follows:

[0145] a′ i =tanh(W′ a ·c′ i +b′ a ) (3-1)

[0146]

[0147] s′ t =Σu′ i c′ i (5-1)

[0148] Among them, W′ c is the weight matrix, b′ c is the bias vector, l′ is the window size of the CNN layer, c′ i is the result of convolution calculation; a′ i is the attention vector of the ith sentence, W′ a and b′ a is used to generate the attention vector a′ i The weight matrix and bias vector of u is the weight vector, a′ j Represents the attention vector of a sentence in the prompt (also contains a′ i ), the maximum value of j is the number of prompt sentences, u′ i is the attention weight; s′ t represents the final feature representation of the tth sentence after the pooling operation of the attention layer, and S represents the set of all sentence representations in the prompt obtained after the pooling operation of the attention layer, that is, S′=[s′1,s′2,…,s′ m ],s′ m Indicates the last sentence of the prompt;

[0149] Step 2-2-2, add a multi-head attention mechanism to capture the relationship between different features of the prompt and improve the accuracy of the score. The corresponding calculation formula is as follows:

[0150]

[0151] Among them, Attention represents the scaling dot product attention mechanism, represents the i-th attention mechanism head, h represents the number of attention heads, represents the last attention mechanism head, is the parameter matrix corresponding to query, key, and value in the multi-head attention mechanism; j represents the feature task to be scored (for example, feature tasks include: overall score, content, word choice, sentence fluency, etc.); Concat means connecting the results of all attention mechanism heads, V jo is the dot product weight matrix;

[0152] Step 2-2-3, perform sentence-level Dropout layer operation to prevent overfitting of the model;

[0153] h2′ i =Dropout(h2 i ) (8-1)

[0154] Among them, h2′ i is the i-th sentence h2 i Feature representation after the Dropout layer;

[0155] Step 2-3, perform article-level encoding representation of prompts: use the long short-term memory network layer LSTM with an attention layer to perform article-level feature representation of prompts; the expression of LSTM is as follows:

[0156]

[0157] The input is the sentence representation [h2′1,h2′2,…h2′ m ], represents the hidden representation of the jth feature scoring task after LSTM with a step length of t; LSTM will automatically traverse all time steps t of this sequence, that is, the value range of step length t is 1-m; h2′ t-1 It represents the sentence representation when the step length is t-1, h2′ t It represents the sentence representation with the current step length of t. The two are used as input to get The application of the LSTM layer enables the model to capture the sequential dependencies and long-range dependencies in the article, so that the model can learn contextual representations and extract useful features; the attention mechanism layer allows the model to pay more attention to the important information in the article. Finally, the final shallow feature representation of the prompt to which the composition belongs is output.

[0158] Further, in a specific embodiment, the prompt-composition correlation feature in step 3 can reflect whether the composition meets the prompt requirements, that is, whether the composition is off-topic, which is also an indispensable scoring point in manual scoring. Therefore, automatic composition scoring cannot ignore its importance. Regarding the prompt-composition correlation feature extraction method: The reason why the present invention proposes to compare the semantics of the prompt with the entire article and its fragments at the same time to extract the prompt-composition correlation feature is that, compared with simply using the entire article to extract features, the method proposed by the present invention is to segment the article according to its meaning before making comparisons. This makes it easier to determine whether the author has included irrelevant content in the article. In addition, this method better reflects the internal thematic coherence of the article. The specific steps are as follows:

[0159] Step 3-1: Without considering the essay prompts, use the Qwen-max large language model launched by Alibaba to perform semantic text segmentation on the essay and divide the input essay into 1-n segments.

[0160] Step 3-2: Use cosine similarity to compare the correlation between the prompt and the entire composition, and between the prompt and the composition fragment, to extract the prompt-composition correlation, as shown in the following formulas (10)-(13):

[0161]

[0162] sim e =cos(essay,prompt) (12)

[0163] sim i =cos(segment i ,prompt) (13)

[0164] Among them, sim e is the cosine similarity between the entire article and the prompt, sim i is the cosine similarity between the current article segment and the prompt, and PR represents the final relevance between the prompt and the article. α represents the reward weight for the relevance between the article and the prompt, which is used to strengthen the influence of the number of article segments on the judgment of the relevance of the prompt. The more segments the essay is divided into, the higher the weight. As the number of segments increases, it eventually approaches the limit value of 1 to reduce the excessive influence of the segments on the article. γ is a random number between -0.03 and +0.03, which is introduced to prevent overfitting of the training data.

[0165] Step 3-3: Considering that the features of the composition itself cannot reflect whether the composition meets the prompt requirements, the present invention connects the features of the composition itself (such as simple features such as the number of words and the number of typos) with the extracted prompt-composition correlation features to form the final feature set fs.

[0166] In a specific embodiment, the feature fusion training prediction module is used to perform feature connection operations on the extracted features and the composition code and prompt code for the final composition scoring. The specific steps are as follows:

[0167] Step 4: Using the prompt-essay attention mechanism can effectively capture the relationship between the prompt and the essay, providing effective support for scoring; the specific steps are as follows:

[0168] Step 4-1: Use a multi-head attention mechanism to implement this article prompt attention mechanism, where the query is the encoded representation of the prompt, and the key and value are the encoded representations of the essay. This setting allows viewing sub-parts of the prompt at each position in the essay. Therefore, the article prompt attention can effectively capture the relationship between the prompt and the essay, providing effective support for scoring.

[0169] Step 4-2: Then connect an LSTM layer and use the average pooling layer and the maximum pooling layer to obtain the prompt article representation. The maximum pooling layer and the average pooling layer are used here because their advantage over the attention pooling layer is that it has fewer parameters when the effect is good. The pooling operation is shown in the following formula:

[0170]

[0171] Among them, x j represents the output of the LSTM layer, and They represent the results after performing the maximum pooling layer operation and the average pooling layer operation, respectively. j yes and The output obtained after the Concat operation represents the final representation of the j-th scoring task.

[0172] Step 4-3: Get the final representation E of the j-th scoring task j Connect with the extracted final feature set fs: I j =[E j ; fs], forming the final input of the model;

[0173] Step 4-4: Then use trait-attention to focus on the representation of other features of the composition. The formula is as follows:

[0174] A=[I 1 ,I 2 ,…,I M ] (17)

[0175]

[0176] r j =[I j ; g j ] (20)

[0177] Among them, A represents the connection of each task, A -j represents the masking of the target composition feature representation, is the attention weight of the ith feature, g j and r j represents the attention vector and the final feature representation.

[0178] Step 4-5: Get the final score through the sigmoid function after the linear layer:

[0179] The technical effect experiment of the present invention illustrates that the experimental details used in the training model are as follows:

[0180] The experimental parameters are set as follows: Dropout rate is 0.5, CNN filter and kernel size are 100 and 5 respectively, LSTM unit is set to 100, embedding dimension is 50, batch_size is 10, the number of heads in the multi-head attention mechanism is 2, the embedding dimension is 100, and RMSprop algorithm is used for optimization. The code is implemented in TensorFlow 2.8.2 and Python 3.7.16, and the graphics card used is Geforce RTX 3090 GPU card.

[0181] The loss functions used in the experiment are MSE and TS. MSE is the most commonly used loss function in automatic scoring tasks; TS is a feature similarity loss function, which is used in cross-scenario scoring tasks and can consider the relationship between different features. The principle of this loss is that when the similarity between the real feature score vector and the predicted feature score vector exceeds a certain threshold, the model learns to improve the similarity of the predicted feature score vector. The formula is as follows:

[0182]

[0183] where y j and k are the jth true feature score vector and the predicted feature score vector, respectively. cos represents cosine similarity, p represents Pearson correlation coefficient, and c represents the number of calculated TSs that are not 0. j starts from 2 because the overall feature (j=1) has a low correlation with other features, so it is excluded. θ is set to 0.7.

[0184] In order to verify whether the automatic essay scoring system proposed in the present invention is effective, the present invention uses the publicly available automatic student assessment award datasets (ASAP and ASAP++) to conduct experiments, as shown in Table 1. In addition, the present invention also uses the most common evaluation indicator: Quadratic Weighted Kappa (QWK) as the evaluation indicator of the present invention. The formula is as follows:

[0185]

[0186] Among them, W i,j represents the quadratic weighted matrix based on the difference between expert ratings and system ratings, E i,j represents the probability product of expert rating i and system rating j.

[0187] Table 1 Dataset information table

[0188]

[0189]

[0190] The scoring system of the present invention was tested and compared with some representative baseline systems. The test results are shown in Tables 2 and 3 below:

[0191] Table 2 Test results

[0192]

[0193] As shown in Table 2, the model proposed in the present invention outperforms almost all baseline models in the scoring task. The model of the present invention outperforms the best baseline model ProTACT by 5.5% in terms of average QWK score. Under different prompts, EPCTS shows significant performance improvements, especially on prompts 4, 7 and 8, with improvements of 8.5%, 20.4% and 10.3%, respectively. The present invention observes that the greatest improvements are concentrated on prompts 7 and 8, which are low-resource prompts with limited training data and contain articles of similar types. Except that the QWK score under prompt 3 is slightly lower than that of ProTACT, the performance under other prompts has improved to varying degrees. These results demonstrate the effectiveness of the model of the present invention in the cross-prompt scenario scoring task.

[0194] Table 3 Test results

[0195]

[0196]

[0197] Among them, Overall: total score; Content: content; Org: writing organization; WC: vocabulary choice; SF: sentence fluency; Conv: convention, such as spelling standards, punctuation usage, etc.; PA: theme adherence; Lang: language; Nar: narrative.

[0198] For the feature scoring dimension (as shown in Table 3), EPCTS outperforms the best baseline model ProTACT, with an average QWK improvement of 5.7%, and achieves the highest scoring performance among all features. We observe that the scores of Content, Org, and Conv are improved by 19.6%, 11.5%, and 11.1%, respectively. The lowest-scoring trait Conv also reaches 0.518. This shows that the method can effectively alleviate the data scarcity problem of specific feature scoring tasks caused by insufficient data.

[0199] The present invention adopts the above technical solution and has the following technical advantages: (1) The present invention improves the accuracy and consistency of AI scoring, which can reduce the pressure of manual scoring and reduce the cost of essay evaluation. (2) The present invention introduces a new method that combines part-of-speech tagging and syntactic features for the coding of prompts, which can more effectively obtain the shallow feature representation of prompts and improve the accuracy of scoring. (3) The present invention designs a new method to calculate the correlation between prompts and articles. The algorithm uses a large language model to perform text semantic segmentation to obtain short article fragments, and combines cosine similarity to obtain the similarity between prompts and short articles and between short article fragments. Thereby, it is possible to better judge whether the composition meets the requirements of the prompt, thereby improving the accuracy of system scoring.

[0200] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.

Claims

1. A method for automatic English composition scoring based on known prompts across scenarios, characterized by: It includes the following steps: Step 1: Extract shallow language features of the composition: adopt the feature representation method of part-of-speech mapping, and obtain the final shallow feature representation of the composition through a series of neural network layers; Step 2: Extract the shallow language features of the prompt: Use a feature representation method that combines part-of-speech tagging and syntactic features, and obtain the final shallow feature representation of the prompt through a series of neural network layers Step 3: Extract the relationship between the prompt and the composition; at the same time, compare the semantics of the prompt with the entire article and the fragment to extract the prompt-composition correlation features to judge whether the composition meets the requirements of the theme; Step 4: Perform feature connection operation on the extracted features, composition code, and prompt code to obtain the final composition score.

2. The method for automatic English composition scoring based on known prompts across scenarios according to claim 1 is characterized by: Step 1 specifically includes the following steps: Step 1-1, performing word-level encoding representation of the composition, specifically includes the following steps: Step 1-1-1, segment the sentence and then perform data cleaning; Step 1-1-2, use the NLTK package in Python to perform part-of-speech tagging on each word, generate a part-of-speech dictionary through statistical methods, and map each word to the corresponding vector space. The word vector sequence after composition mapping is (P1, P1, ..., P i ,…,P n ), P i is the dense vector of the i-th word in the composition, and n is the total number of words; Step 1-1-3, use the word-level Dropout layer to randomly discard a certain percentage of neuron outputs to obtain the feature representation P′ i , reducing the model's dependence on specific neurons and improving the model's generalization ability. The specific expression is as follows: P′ i =Dropout(P i ) (1) Among them, P′ i It is the feature representation after the Dropout layer; Step 1-2, perform sentence-level encoding of the composition: specifically includes the following steps: Step 1-2-1, represent the feature P′ i Learn low-level feature representations useful for the scoring task through a shared 1D convolutional encoder with an attention layer; The expression of the shared 1D convolutional encoder is as follows: c i =f(W c ·[P′ i :P′ i+l-1 ]+b c ) (2) The expression of the attention mechanism's processing method is as follows: a i =tanh(W a ·c i +b a ) (3) s t =Σu i c i (5) Among them, W c is the weight matrix, b c is the bias vector, l is the window size of the CNN layer, c i is the result of convolution calculation; a i is the attention vector of the current i-th sentence, W a and b a It is used to generate the current attention vector a i The weight matrix and bias vector of ; Indicates the attention of the current sentence; The sum represents the attention sum of all sentences in the composition, w u is the weight vector, a j represents the attention vector of any sentence in the composition, and the maximum value of j is the number of sentences in the composition; u i is the attention weight; s t represents the final feature representation of the tth sentence after the pooling operation of the attention layer, and S represents the set of all sentence representations in the composition obtained after the pooling operation of the attention layer, that is, S = [s1, s2, ..., s m ],s m Indicates the last sentence of a composition; Step 1-2-2, add a multi-head attention mechanism to capture the relationship between different features of the composition and improve the accuracy of the score. The corresponding calculation formula is as follows: Among them, Attention represents the scaling dot product attention mechanism, represents the i-th attention mechanism head, h represents the number of attention heads, represents the last attention mechanism head, is the parameter matrix corresponding to query, key, and value in the multi-head attention mechanism; j represents the feature task to be scored on the current composition; Concat means connecting the results of all attention mechanism heads, W jo is the dot product weight matrix; Step 1-2-3, assume that the sentence representation set after the multi-head attention mechanism is H = mhead(S) j ,[h1,h2,…,h m ] represent the representation of each sentence in the composition, and then perform sentence-level Dropout layer operation to prevent overfitting of the model; h′ i =Dropout(h i ) (8) Among them, h′ i is the i-th sentence h i Feature representation after the Dropout layer; Steps 1-3, perform article-level encoding representation of the composition: use the long short-term memory network layer LSTM with an attention layer to perform article-level feature representation; the expression of LSTM is as follows: The input is a sentence representation [h′1,h′2,…h′ m ], represents the hidden representation of the jth feature scoring task after LSTM with a step length of t; LSTM will automatically traverse all time steps t of this sequence, that is, the value range of step length t is 1-m; h′ t-1 It represents the sentence representation when the step length is t-1, h′ t It represents the sentence representation with the current step length of t. The two are used as input to get 3. The method for automatic English composition scoring based on known prompts across scenarios according to claim 2 is characterized by: In step 1-2-2, the number of attention heads is set to 2.

4. The method for automatic English composition scoring based on known prompts across scenarios according to claim 1 is characterized in that: Step 2 specifically includes the following steps: Step 2-1, performing word-level encoding representation of the prompt; specifically comprising the following steps: Step 2-1-1, segment the sentence and then perform data cleaning; Step 2-1-2, use the NLTK package in Python to perform part-of-speech tagging on each word, generate a part-of-speech dictionary through statistical methods, and map each word to the corresponding vector space. The sequence of part-of-speech features after the mapping is (tops1, tops2, …, tops i ,…,tops n ); At the same time, the Spacy package in Python is used to extract the syntactic features of each word, and a syntactic dictionary is generated through statistical methods, and each word is mapped to the corresponding vector space. The syntactic feature sequence after the mapping is (tsyn1, tsyn2, …, tsyn i ,…,tsyn n ); Step 2-1-3, add the part-of-speech feature sequence and syntactic feature sequence after prompt mapping element by element to obtain the mapping Q of each word in the prompt in the vector space i , the calculation formula is Q i =Add(tpos i ,tsyn i ); In step 2-1-4, a certain percentage of neuron outputs are randomly discarded using a word-level Dropout layer to reduce the model’s reliance on specific neurons, thereby improving the model’s generalization ability. The formula is as follows (1-1): Q′ i =Dropout(Q i ) (1-1) Among them, Q i The dense vector representing the i-th word of the prompt, Q i ′ is the feature representation after the Dropout layer; Step 2-2, performing sentence-level encoding representation of the prompt; specifically comprising the following steps: Step 2-2-1, Q i ′ is fed into a shared 1D convolutional encoder with an attention layer to learn low-level feature representations useful for the scoring task; The expression of the shared 1D convolutional encoder is as follows: c′ i =f(W′ c ·Q′ i :Q′ i+l′-1 ]+b′ c ) (2-1) The expression of the attention mechanism's processing method is as follows: a' i =tanh(W′ a ·c' i +b′ a ) (3-1) s′ t =Σu′ i c′ i (5-1) Among them, W′ c is the weight matrix, b′ c is the bias vector, l′ is the window size of the CNN layer, c′ i is the result of convolution calculation; a′ i is the attention vector of the ith sentence, W′ a and b′ a is used to generate the attention vector a′ i The weight matrix and bias vector of u is the weight vector, a′ j represents the attention vector of a sentence in the prompt, the maximum value of j is the number of prompt sentences, u′ i is the attention weight; s′ t represents the final feature representation of the tth sentence after the pooling operation of the attention layer, and S represents the set of all sentence representations in the prompt obtained after the pooling operation of the attention layer, that is, S′=[s′1,s′2,…,s′ m ],s′ m Indicates the last sentence of the prompt; Step 2-2-2, add a multi-head attention mechanism to capture the relationship between different features of the prompt and improve the accuracy of the score. The corresponding calculation formula is as follows: Among them, Attention represents the scaling dot product attention mechanism, represents the i-th attention mechanism head, h represents the number of attention heads, represents the last attention mechanism head, is the parameter matrix corresponding to query, key, and value in the multi-head attention mechanism; j represents the feature task to be scored; Concat means connecting the results of all attention mechanism heads, V jo is the dot product weight matrix; Step 2-2-3, assume that the sentence representation set after the multi-head attention mechanism is H2 = mhead(S′) j ,[h21,h22,…,h2 m ] represent the representation of each sentence in the prompt, and then perform sentence-level Dropout layer operation to prevent overfitting of the model; h2′ i =Dropout(h2 i ) (8-1) Among them, h2′ i is the i-th sentence h2 i Feature representation after the Dropout layer; Step 2-3, perform article-level encoding representation of prompts: use the long short-term memory network layer LSTM with an attention layer to perform article-level feature representation; the expression of LSTM is as follows: The input is the sentence representation [h2′1,h2′2,…h2′ m ], represents the hidden representation of the jth feature scoring task after LSTM with a step length of t; LSTM will automatically traverse all time steps t of this sequence, that is, the value range of step length t is 1-m; h2′ t-1 It represents the sentence representation when the step length is t-1, h2′ t It represents the sentence representation with the current step length of t. The two are used as input to get 5. The method for automatic English composition scoring based on known prompts across scenarios according to claim 1 is characterized by: In step 3, the article is first segmented according to meaning to determine whether the author has included irrelevant content in the article.

6. The method for automatic English composition scoring based on known prompts across scenarios according to claim 1 is characterized by: Step 3 specifically includes the following steps: Step 3-1, without considering the writing prompt, use the existing mature large language model to perform semantic text segmentation on the composition, and divide the input composition into 1-n segments; Step 3-2, using cosine similarity to compare the correlation between the prompt and the entire composition, and between the prompt and the composition fragment, to calculate the prompt-composition correlation. The specific calculation formula is as follows: sim e =cos(essay,prompt) (12) sim i =cos(segment i ,prompt) (13) Among them, sim e is the cosine similarity between the entire article and the prompt, sim i is the cosine similarity between the current article segment and the prompt, PR represents the final relevance between the prompt and the article; α represents the reward weight of the relevance between the article and the prompt, which is used to strengthen the influence of the number of article segments on the judgment of prompt relevance; γ is a random number between -0.03 and +0.03 to prevent overfitting of the training data; Step 3-3, connect the composition features and the extracted prompt-composition correlation features to form the final feature set fs.

7. The method for automatic English composition scoring based on known prompts across scenarios according to claim 1 is characterized by: Step 4 uses the prompt-essay attention mechanism to capture the relationship between prompts and essays, providing effective support for scoring; the specific steps are as follows: Step 4-1, use the multi-head attention mechanism to implement the article prompt attention mechanism, where query is the encoded representation of the prompt, and key and value are the encoded representations of the essay; Step 4-2, then connect an LSTM layer, and use the average pooling layer and the maximum pooling layer to obtain the prompt article representation; the formula for the pooling operation is as follows: Among them, x j represents the output of the LSTM layer, and They represent the results after performing the maximum pooling layer operation and the average pooling layer operation, respectively. j yes and The output obtained after the Concat operation represents the final representation of the j-th scoring task; Step 4-3: Get the final representation E of the j-th scoring task j Connect with the extracted final feature set fs: I j =[E j ; fs], forming the final input of the model; Step 4-4, then use feature attention to focus on the representation of other features of the composition. The formula is as follows: A=[I 1 ,I 2 ,…,I j ,…,I M ] (17) r j =[I j ;g j ] (20) Among them, A represents the connection of each task, A -j represents the masking of the target composition feature representation, is the attention weight of the ith feature, g j and r j Represents the attention vector and the final feature representation; Step 4-5, the final score is obtained by the sigmoid function after the linear layer:

8. The method for automatic English composition scoring based on known prompts across scenarios according to claim 2 or 4, characterized in that: Feature tasks included total score, content, word choice, and sentence fluency.

9. A system for automatically scoring English essays across scenarios based on known prompts, which adopts the method for automatically scoring English essays across scenarios based on known prompts as claimed in any one of claims 1 to 8, characterized in that: The system includes: Composition shallow feature extraction module: extracts the shallow language features of the composition, namely, part-of-speech features; Used to encode the essays that need to be scored and obtain shallow feature representations of the essays; Hint shallow feature extraction module: extracts the shallow language features of the hint, which include grammatical features and syntactic features; It is used to encode the prompt to which the essay to be graded belongs and obtain the shallow feature representation of the prompt; Prompt-composition correlation extraction module: extracts the relationship between prompts and compositions to judge whether the compositions meet the requirements of the theme; feature fusion training prediction module: performs feature connection operations on the extracted features and composition codes and prompt codes for the final composition scoring.

Images