A large-model junior middle school English cloze generation method based on self-supervised learning

Abstract

The invention discloses a method for generating a large-scale junior high school English cloze test based on self-supervised learning, comprising S1, collecting English text data as a training data set and cloze test questions of the senior high school entrance examination in various provinces and cities within 10 years as a text adaptation learning data set; S2, forming a question vocabulary list; S3, and pre-processing the training data set; S4, using a self-supervised learning method to build a large-scale natural language processing model; S5, fine-tuning the large-scale natural language processing model according to the training data set and the text adaptation learning data set; S6, automatically generating junior high school English cloze test questions based on the trained large-scale natural language processing model and the adapted text; S7, automatically checking and manually reviewing the generated cloze test questions; S8, storing the generated cloze test questions in a question bank. The invention utilizes self-supervised learning and a large-scale natural language processing model to efficiently generate a large number of cloze test questions, greatly saving manpower and time costs.

Description

Technical Field

[0001] The present invention relates to the field of English, and in particular to a large-model junior high school English cloze generation method based on self-supervised learning. Background Art

[0002] With the development of educational technology and the increasing demand for intelligent teaching, the automatic generation of high-quality English cloze questions has become an important research direction. Traditional cloze question generation methods mainly rely on manual writing or simple rule-based generation methods, which have many limitations in practice.

[0003] First, manually writing cloze questions takes a lot of time and effort. Each question requires teachers to carefully design, including selecting appropriate text paragraphs, determining blocked words, and designing reasonable options. This method is obviously difficult to meet large-scale educational needs. Especially in junior high school, students need a lot of practice questions to improve their English level, and it is difficult to provide a sufficient number and diversity of questions by relying on manual writing.

[0004] Secondly, although simple rule-based methods can improve the efficiency of question generation to a certain extent, these methods often ignore the complexity and diversity of language, and the quality of the generated questions varies. For example, some simple generation algorithms based on word frequency or sentence patterns cannot accurately grasp the semantics and contextual relationships of sentences, resulting in the lack of logic and consistency in the generated questions. In addition, these methods are difficult to adjust according to students' learning level and personalized needs, and cannot provide targeted practice questions.

[0005] The natural language processing (NLP) technology in the prior art provides some new ideas for automatically generating English cloze questions. For example, the text generation method based on pre-trained language models (such as GPT, BERT, etc.) can generate more natural text paragraphs through training with large-scale corpora. However, the existing pre-trained models still have some shortcomings in generating cloze questions. First, these models require a large amount of annotated data for fine-tuning, and the acquisition cost of annotated data is high and difficult to obtain. Secondly, when generating questions, the existing models lack special optimization of educational content, and the generated questions may not meet the teaching needs, especially in junior high school English teaching, which has special requirements for the mastery of vocabulary and grammar. Therefore, how to provide a large model junior high school English cloze generation method based on self-supervised learning is a problem that technicians in this field urgently need to solve. Summary of the invention

[0006] One purpose of the present invention is to propose a large-model junior high school English cloze generation method based on self-supervised learning. The present invention utilizes self-supervised learning and a large-scale natural language processing model to efficiently generate a large number of cloze questions, greatly saving manpower and time costs.

[0007] A method for generating a large-model junior high school English cloze test based on self-supervised learning according to an embodiment of the present invention comprises the following steps:

[0008] S1. Collect a large amount of unlabeled English text data as training data set and cloze test questions from various provinces and cities in the past 10 years as text adaptation learning data set;

[0009] S2. Collect and organize the vocabulary of various junior high school English textbooks and arrange them according to the parts of speech and meanings of the unit lessons and the key words in bold to form a vocabulary list for questions;

[0010] S3, and preprocessing the training data set, wherein the preprocessing includes removing noise data, segmenting words, and marking parts of speech;

[0011] S4. Use a self-supervised learning method in combination with a preprocessed training data set to build a large-scale natural language processing model, and pre-train the large-scale natural language processing model, wherein the self-supervised learning method includes an occlusion prediction task and a contrastive learning task, wherein the occlusion prediction task includes randomly occluding some words in English text data, and allowing the large-scale natural language processing model to predict the occluded words, and the contrastive learning task includes constructing positive and negative sample pairs, and training by maximizing the similarity of the positive sample pairs and minimizing the similarity of the negative sample pairs;

[0012] S5. Fine-tune the large-scale natural language processing model according to the training data set and the text adaptation learning data set. The task fine-tuning includes applying the pre-trained large-scale natural language processing model to the adaptation of English cloze text materials and the generation of questions, and further optimizing the large-scale natural language processing model through a supervised learning method so that it can generate cloze questions that meet the needs of junior high school English teaching;

[0013] S6. Automatically generate junior high school English cloze questions based on the trained large-scale natural language processing model and the adapted text, including selecting appropriate text paragraphs as basic materials, and randomly selecting a number of words in the text based on the question vocabulary list for masking, to generate cloze questions with options;

[0014] S7, automatically checking and manually reviewing the generated cloze questions to ensure that the difficulty and content of the questions are suitable for the learning needs of junior high school students, wherein the automatic checking includes checking the grammatical correctness and semantic rationality of the generated questions, and the manual review includes evaluating and modifying the questions by English teaching experts;

[0015] S8. The generated cloze test questions are stored in a question bank, and the questions are dynamically adjusted and distributed according to the learning data and performance of the junior high school students.

[0016] Optionally, the S1 includes the following steps:

[0017] S11, collecting a large amount of unlabeled English text data, where the sources of the English text data include online articles, books, news reports and conversation records;

[0018] S12. Clean and deduplicate the collected English text data to remove noise data and duplicate data:

[0019] D clean ={d i ∈D|if score(d i )≥λ};

[0020] Where D represents the initial data set, d i represents the i-th data in the data set, score(d i ) represents data d i The noise score, λ is the threshold for removing noise, D clean represents the cleaned data set;

[0021] S13. Perform word segmentation on the cleaned English text data, split each sentence into word sequences, and perform part-of-speech tagging on each word to generate a word segmentation text dataset with part-of-speech tagging:

[0022] {(w1,t1),(w2,t2),...,(w n , t n )}=POS_Tagger(Tokenikzer(D clean ));

[0023] Among them, w i represents the i-th word, t i Represents the part-of-speech tag of the i-th word, Tokenizer is the word segmentation algorithm, and POS_Tagger is the part-of-speech tagging algorithm;

[0024] S14. Collect the cloze test papers of the senior high school entrance examinations in various provinces and cities in the past 10 years, including the cloze test papers of the senior high school entrance examinations in previous years;

[0025] S15. Sort and annotate the collected cloze test questions, match the text paragraph of each question with the corresponding correct option, and mark the test text as a text adaptation learning dataset:

[0026] Q prep ={(q i , A i )|q i ∈Q,A i =Match(q i )};

[0027] Among them, Q represents the collection of real questions collected, q i represents the i-th question, A i represents the correct option for the i-th question, Match is the matching function, Q prep Represents the sorted real-question dataset;

[0028] S16, construct a training data set and a text adaptation learning data set, and save the part-of-speech annotated English text data after word segmentation and the high school entrance examination cloze test text adaptation learning data set separately:

[0029] Dataset train ={{(w1,t1), (w2,t2),..., (w n , t n )}};

[0030] Dataset adapt =Q prep ;

[0031] Among them, Dataset train Represents the training data set after word segmentation and part-of-speech tagging, Dataset adapt Represents the sorted real test text adaptation learning dataset.

[0032] Optionally, S2 includes the following steps:

[0033] S21, collecting vocabulary from various versions of English textbooks for junior high schools, including unit vocabulary, class vocabulary and boldface key vocabulary in the textbooks, to form a preliminary vocabulary list;

[0034] S22, sorting the preliminary vocabulary list, classifying the vocabulary according to units and classes, and marking the part of speech and meaning of each vocabulary word to generate a classified and annotated vocabulary list:

[0035] V unit ={(w i , p i , m i )|w i ∈W,p i=POS(w i ), m i =Meanimg(w i )};

[0036] Among them, V unit represents a vocabulary classified by unit, w i represents the i-th word, p i Represents vocabulary w i Part of speech, m i Represents vocabulary w i POS is the part-of-speech tagging function, and Meaning is the meaning tagging function;

[0037] S23, extracting the boldface key words from the classification and annotation vocabulary list to form a boldface key words list:

[0038] V bold ={w i ∈V unit |if isBold(w i )};

[0039] Among them, V bold represents the bold emphasis vocabulary list, isBold is a function to determine whether a word is a bold emphasis word;

[0040] S24. Sort and arrange the classified and annotated vocabulary lists and the boldface key vocabulary lists according to units and lessons to form the final vocabulary list for questions:

[0041] V final ={(w i , p i , m i )∈V unit ∪V bola};

[0042] Among them, V final Indicates the final vocabulary list for the test, including classified and marked vocabulary and bold key vocabulary.

[0043] Optionally, S4 includes the following steps:

[0044] S41. Use the preprocessed English text data Dataset train Constructing a large-scale natural language processing model, wherein the large-scale natural language processing model includes an input layer, an encoding layer, a self-supervised learning task layer, and an output layer;

[0045] S42. In the self-supervised learning task layer, an occlusion prediction task is designed to randomly occlude some words in the input text data and let the model predict the occluded words. The loss function of the occlusion prediction task is defined as:

[0046]

[0047] in, is the loss function of the occlusion prediction task, N is the number of occluded words, P(w i |w \i ) means that in the given context w \i Predict the occluded word w under the condition i probability;

[0048] S43. Design a contrastive learning task, construct positive and negative sample pairs, and train by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The loss function of the contrastive learning task is defined as:

[0049]

[0050] in, is the loss function of the contrastive learning task, N is the number of positive sample pairs, M is the number of negative sample pairs, and h i represents the vector of the i-th sample, Represents the vector of the i-th positive sample pair, the positive sample pair vector is the same as h i The feature vectors of similar sample pairs, Represents the vector of the jth negative sample pair, the negative sample pair vector is the same as h i The feature vectors of dissimilar sample pairs, Represents sample h i and positive sample pairs The similarity between Represents sample h i And negative sample pairs The similarity between

[0051] S44. Design a sentence order prediction task to let the large-scale natural language processing model determine whether two sentences are arranged in the order of the original text. The loss function of the sentence order prediction task is defined as:

[0052]

[0053] in, is the loss function for the task of predicting the order of sentences, N is the number of sentence pairs, y i Indicates whether the i-th sentence pair is in the correct order, P(y i |s i ,s i+1 ) represents the probability of the model judging the order of the i-th sentence pair;

[0054] S45. Pre-train a large-scale natural language processing model by combining the occlusion prediction task, contrastive learning task and sentence order prediction task. The total loss function of the pre-training is defined as:

[0055]

[0056] in, is the total loss function, λ1, λ2, λ3 are adaptive weight parameters used to dynamically adjust the impact of each task loss.

[0057] Optionally, S5 includes the following steps:

[0058] S51. According to the training data set Dataset train and text adaptation learning dataset adapt Fine-tuning the pre-trained large-scale natural language processing model, wherein the task fine-tuning includes applying the pre-trained model to the task of adapting and generating text materials for English cloze questions;

[0059] S52, training data set Dataset train Each sentence in s i and the corresponding annotated part-of-speech vocabulary as input to generate a preliminary cloze test question:

[0060] Q i ={(s i ,{(w i1 , t i1 ), (w i2 , t i2 ),...,(w in , t in )})};

[0061] Among them, Q i represents the i-th generated preliminary cloze question, s i For a sentence, {(w i1 , t i1 ),(w i2 , t i2 ),...,(w in , t in )} is a vocabulary and part-of-speech tag pair;

[0062] S53. Adapt the preliminary cloze test questions and combine them with the text to adapt the learning dataset adapt The passage in the middle school entrance examination cloze test real test text {q i , A i}, adjust the difficulty and content of the question, and generate the final cloze question:

[0063] Qfinal ={(q i , {A i},{(w i1 , t i1 ), (w i2 , t i2 ),...,(w in , t in )})};

[0064] Among them, Q final represents the final generated cloze question, q i For the real test text paragraph, A i is the corresponding correct option;

[0065] S54. Further optimize the large-scale natural language processing model through supervised learning methods, use the training dataset and the adapted cloze question dataset for fine-tuning training, and optimize the model parameters to minimize the loss between the generated questions and the real questions:

[0066]

[0067] in, is the loss function of fine-tuning training, N is the number of training samples, Q final (i) is the i-th generated cloze question, is the i-th real question;

[0068] The fine-tuned large-scale natural language processing model can generate cloze questions that meet the needs of junior high school English teaching.

[0069] Optionally, the S6 comprises the following steps:

[0070] S61. Select appropriate text paragraphs as basic materials based on the trained large-scale natural language processing model, where the text paragraphs come from the preprocessed English text data Dataset train and text adaptation learning dataset adapt , the formula for selecting the appropriate text paragraph is:

[0071]

[0072] Among them, score(d i ) indicates paragraph d i The score of , λ is the selection threshold, length(d i ) represents a text paragraph d i The length of freq(l j ) indicates the length l j The frequency distribution of , k is the number of candidate lengths;

[0073] S62, the selected text paragraph is segmented into several sentences. The sentence segmentation formula is:

[0074]

[0075] Among them, split(d i ) means to add the text paragraph d i Split into a collection of sentences;

[0076] S63, based on the vocabulary list V final In each sentence, a number of words are randomly selected for masking to generate cloze questions. The masking selection formula is:

[0077] Q i ={(s i ,{w ij |w ij ∈V final and random() <p})|s i ∈{s1,s2,...,s n}};

[0078] Among them, Q i represents the generated cloze question, w ij is the occluded word, random() is the random function, and p is the occlusion probability;

[0079] S64. Generate several options for each blocked word, including a correct option and several interference options, where the interference options are selected from words with the same part of speech and similar meanings:

[0080] A ij ={w jz ,{w ijm |w ijm ∈V final and POS(w ijm )=POS(w ij )and sim(w ij , w ijm )≥δ}};

[0081] Among them, A ij represents a set of options, w jz is the correct option, w ijm , POS(w ijm ) is the part-of-speech tag, sim(w ij , w ijm ) is the similarity function, δ is the similarity threshold;

[0082] S65. Combine the generated sentences, the blocked words and their options to form the final cloze test question:

[0083] Q final ={(s i ,{(w ij , A ij )})|(s i ,{w ij})∈Q i};

[0084] Among them, Q final Represents the final generated cloze test question, including the sentence, the masked words and their corresponding option sets.

[0085] The beneficial effects of the present invention are:

[0086] (1) The present invention uses self-supervised learning and large-scale natural language processing models to efficiently generate a large number of cloze test questions, greatly saving manpower and time costs. When generating questions, a pre-processed large-scale data set and a fine-tuned model are used to generate questions that meet the needs of junior high school English teaching. The combination of occlusion prediction tasks and contrastive learning tasks makes the generated questions more accurate in grammar and semantics, and the quality and consistency are effectively guaranteed.

[0087] (2) The present invention realizes personalized teaching content push by dynamically adjusting and distributing the generated questions according to the students' learning data and performance. The model can automatically generate questions suitable for their learning stage according to the learning level and needs of different students, providing intelligent educational support.

[0088] (3) The method of the present invention is applicable to various junior high school English textbook versions. By collecting and organizing the vocabulary in the textbooks, a vocabulary list for questions is generated to ensure that the generated questions are closely integrated with the teaching content. At the same time, based on the pre-training and fine-tuning of a large-scale natural language processing model, the method has high flexibility and can adapt to the teaching needs of different regions and different textbook versions. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0090] Figure 1 A flowchart of a method for generating a large-model junior high school English cloze test based on self-supervised learning proposed by the present invention;

[0091] Figure 2The present invention provides a flowchart for masking cloze questions and generating options based on a question vocabulary list in a large-model junior high school English cloze generation method based on self-supervised learning proposed by the present invention. DETAILED DESCRIPTION

[0092] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0093] refer to Figure 1-2 , a large model junior high school English cloze generation method based on self-supervised learning, comprising the following steps:

[0094] S1. Collect a large amount of unlabeled English text data as training data set and cloze test questions from various provinces and cities in the past 10 years as text adaptation learning data set;

[0095] S2. Collect and organize the vocabulary of various junior high school English textbooks and arrange them according to the parts of speech and meanings of the unit lessons and the key words in bold to form a vocabulary list for questions;

[0096] S3, and preprocess the training data set, including removing noise data, segmenting words and marking parts of speech;

[0097] S4. Use a self-supervised learning method in combination with a preprocessed training data set to build a large-scale natural language processing model, and pre-train the large-scale natural language processing model. The self-supervised learning method includes an occlusion prediction task and a contrastive learning task. The occlusion prediction task includes randomly occluding some words in the English text data and letting the large-scale natural language processing model predict the occluded words. The contrastive learning task includes constructing positive and negative sample pairs, and training by maximizing the similarity of the positive sample pairs and minimizing the similarity of the negative sample pairs.

[0098] S5. Fine-tune the large-scale natural language processing model based on the training data set and the text adaptation learning data set. The task fine-tuning includes applying the pre-trained large-scale natural language processing model to the adaptation of English cloze text materials and the generation of questions, and further optimizing the large-scale natural language processing model through supervised learning methods so that it can generate cloze questions that meet the needs of junior high school English teaching;

[0099] S6. Automatically generate junior high school English cloze questions based on the trained large-scale natural language processing model and the adapted text, including selecting appropriate text paragraphs as basic materials, and randomly selecting a number of words in the text based on the question vocabulary list for masking, to generate cloze questions with options;

[0100] S7. Automatically check and manually review the generated cloze questions to ensure that the difficulty and content of the questions are suitable for the learning needs of junior high school students. Automatic checking includes checking the grammatical correctness and semantic rationality of the generated questions. Manual review includes evaluation and modification of the questions by English teaching experts.

[0101] S8. The generated cloze test questions are stored in a question bank, and the questions are dynamically adjusted and distributed according to the learning data and performance of the junior high school students.

[0102] In this implementation, S1 includes the following steps:

[0103] S11. Collect a large amount of unlabeled English text data, including online articles, books, news reports and conversation records;

[0104] S12. Clean and deduplicate the collected English text data to remove noise data and duplicate data:

[0105] D clean ={d i ∈D|if score(d i )≥λ};

[0106] Where D represents the initial data set, d i represents the i-th data in the data set, score(d i ) represents data d i The noise score, λ is the threshold for removing noise, D clean represents the cleaned data set;

[0107] S13. Perform word segmentation on the cleaned English text data, split each sentence into word sequences, and perform part-of-speech tagging on each word to generate a word segmentation text dataset with part-of-speech tagging:

[0108] {{(w1,t1), (w2,t2),..., (w n , t n )}=POS_Tagger(Tokenizer(D clean ));

[0109] Among them, w i represents the i-th word, t i Represents the part-of-speech tag of the i-th word, Tokenizer is the word segmentation algorithm, and POS_Tagger is the part-of-speech tagging algorithm;

[0110] S14. Collect the cloze test papers of the senior high school entrance examinations in various provinces and cities in the past 10 years, including the cloze test papers of the senior high school entrance examinations in previous years;

[0111] S15. Sort and annotate the collected cloze test questions, match the text paragraph of each question with the corresponding correct option, and mark the test text as a text adaptation learning dataset:

[0112] Q prep ={(q i , A i )|q i ∈Q,A i =Match(q i};

[0113] Among them, Q represents the collection of real questions collected, q i represents the i-th question, A i represents the correct option for the i-th question, Match is the matching function, Q prep Represents the sorted real-question dataset;

[0114] S16, construct a training data set and a text adaptation learning data set, and save the part-of-speech annotated English text data after word segmentation and the high school entrance examination cloze test text adaptation learning data set separately:

[0115] Dataset train ={{(w1,t1), (w2,t2),..., (w n , t n )}};

[0116] Dataset adapt =Q prep ;

[0117] Among them, Dataset train Represents the training data set after word segmentation and part-of-speech tagging, Dataset adapt Represents the sorted real test text adaptation learning dataset.

[0118] In this implementation, S2 includes the following steps:

[0119] S21. Collect vocabulary from various versions of junior high school English textbooks, including unit vocabulary, class vocabulary and bold key vocabulary in the textbooks, and form a preliminary vocabulary list;

[0120] S22. Sort out the preliminary vocabulary list, classify the vocabulary according to units and lessons, and mark the part of speech and meaning of each word to generate a classified and annotated vocabulary list:

[0121] V unit ={(w i , p i , m i )|w i ∈W,p i=POS(w i ), m i =Meaning(w i )};

[0122] Among them, V unit represents a vocabulary classified by unit, w i represents the i-th word, p i Represents vocabulary w i Part of speech, m i Represents vocabulary w i POS is the part-of-speech tagging function, and Meaning is the meaning tagging function;

[0123] S23, extracting the boldface key words from the classification and annotation vocabulary list to form a boldface key words list:

[0124] V bold ={w i ∈V unit |if isBold(wi)};

[0125] Among them, V bold represents the bold emphasis vocabulary list, isBold is a function to determine whether a word is a bold emphasis word;

[0126] S24. Sort and arrange the classified and annotated vocabulary lists and the boldface key vocabulary lists according to units and lessons to form the final vocabulary list for questions:

[0127] V final ={(w i , p i , m i )∈V unit ∪V bold};

[0128] Among them, V final Indicates the final vocabulary list for the test, including classified and marked vocabulary and bold key vocabulary.

[0129] In this implementation, S4 includes the following steps:

[0130] S41. Use the preprocessed English text data Dataset train Build a large-scale natural language processing model, which includes an input layer, an encoding layer, a self-supervised learning task layer, and an output layer;

[0131] S42. In the self-supervised learning task layer, an occlusion prediction task is designed to randomly occlude some words in the input text data and let the model predict the occluded words. The loss function of the occlusion prediction task is defined as:

[0132]

[0133] in, is the loss function of the occlusion prediction task, N is the number of occluded words, P(w i |w \i ) means that in the given context w \i Predict the occluded word w under the condition i probability;

[0134] S43. Design a contrastive learning task, construct positive and negative sample pairs, and train by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The loss function of the contrastive learning task is defined as:

[0135]

[0136] in, is the loss function of the contrastive learning task, N is the number of positive sample pairs, M is the number of negative sample pairs, and h i represents the vector of the i-th sample, Represents the vector of the i-th positive sample pair, the positive sample pair vector is the same as h i The feature vectors of similar sample pairs, Represents the vector of the jth negative sample pair, the negative sample pair vector is the same as h i The feature vectors of dissimilar sample pairs, Represents sample h i and positive sample pairs The similarity between Represents sample h i And negative sample pairs The similarity between

[0137] S44. Design a sentence order prediction task to let the large-scale natural language processing model determine whether two sentences are arranged in the order of the original text. The loss function of the sentence order prediction task is defined as:

[0138]

[0139] in, is the loss function for the task of predicting the order of sentences, N is the number of sentence pairs, y i Indicates whether the i-th sentence pair is in the correct order, P(y i |s i ,s i+1 ) represents the probability of the model judging the order of the i-th sentence pair;

[0140] S45. Pre-train a large-scale natural language processing model by combining the occlusion prediction task, contrastive learning task and sentence order prediction task. The total loss function of the pre-training is defined as:

[0141]

[0142] in, is the total loss function, λ1,λ2,λ3 are adaptive weight parameters used to dynamically adjust the impact of each task loss.

[0143] In this implementation, S5 includes the following steps:

[0144] S51. According to the training data set Dataset train and text adaptation learning dataset adapt Fine-tune the pre-trained large-scale natural language processing model for tasks, including applying the pre-trained model to the adaptation and generation tasks of English cloze text materials;

[0145] S52, training data set Dataset train Each sentence in s i and the corresponding annotated part-of-speech vocabulary as input to generate a preliminary cloze test question:

[0146] Q i ={(s i ,{(w i1 , t i1 ),(w i2 ,t i2 ),...,(w in , t in )})};

[0147] Among them, Q i represents the i-th generated preliminary cloze question, s i For a sentence, {(w i1 ,t i1 ),(w i2 ,t i2 ),…,(w in ,t in )} is a vocabulary and part-of-speech tag pair;

[0148] S53. Adapt the preliminary cloze test questions and combine them with the text to adapt the learning dataset adapt The passage in the middle school entrance examination cloze test real question text {q i ,A i}, adjust the difficulty and content of the question, and generate the final cloze question:

[0149] Q final ={(q i , {A i},{(w i1 , ti1 ), (w i2 , t i2 ),...,(w in , t in )})};

[0150] Among them, Q final represents the final generated cloze question, q i For the real test text paragraph, A i is the corresponding correct option;

[0151] S54. Further optimize the large-scale natural language processing model through supervised learning methods, use the training dataset and the adapted cloze question dataset for fine-tuning training, and optimize the model parameters to minimize the loss between the generated questions and the real questions:

[0152]

[0153] in, is the loss function of fine-tuning training, N is the number of training samples, Q final (i) is the i-th generated cloze question, is the i-th real question;

[0154] The fine-tuned large-scale natural language processing model can generate cloze questions that meet the needs of junior high school English teaching.

[0155] In this implementation, S6 includes the following steps:

[0156] S61. Select appropriate text paragraphs as basic materials based on the trained large-scale natural language processing model. The text paragraphs come from the preprocessed English text data dataset. train and text adaptation learning dataset adapt , the formula for selecting the appropriate text paragraph is:

[0157]

[0158] Among them, score(d i ) indicates paragraph d i The score of , λ is the selection threshold, length(d i ) represents a text paragraph d i The length of freq(l j ) indicates the length l j The frequency distribution of , k is the number of candidate lengths;

[0159] S62, the selected text paragraph is segmented into several sentences. The sentence segmentation formula is:

[0160]

[0161] Among them, split(d i ) means to add the text paragraph d i Split into a collection of sentences;

[0162] S63, based on the vocabulary list V final In each sentence, a number of words are randomly selected for masking to generate cloze questions. The masking selection formula is:

[0163] Q i ={(s i ,{w ij |w ij ∈V final and random() <p})|s i ∈{s1,s2,...,s n}}

[0164] Among them, Q i represents the generated cloze question, w ij is the occluded word, random() is the random function, and p is the occlusion probability;

[0165] S64. Generate several options for each blocked word, including a correct option and several interference options, where the interference options are selected from words with the same part of speech and similar meanings:

[0166]

[0167] Among them, A ij represents a set of options, w jz is the correct option, w ijm , POS(w ijm ) is the part-of-speech tag, sim(w ij ,w ijm ) is the similarity function, δ is the similarity threshold;

[0168] S65. Combine the generated sentences, the blocked words and their options to form the final cloze test question:

[0169] Q final ={(s i ,{(w ij ,A ij )})|(s i ,{w ij})∈Q i};

[0170] Among them, Q finalRepresents the final generated cloze test question, including the sentence, the masked words and their corresponding option sets.

[0171] In view of the defects of the prior art, this embodiment proposes a large-model junior high school English cloze generation method based on self-supervised learning, which aims to use self-supervised learning technology and large-scale natural language processing models, through pre-training and fine-tuning, so that the model has the ability to understand and generate natural language texts, and can automatically generate high-quality cloze questions that meet the needs of junior high school English teaching. This method greatly reduces the need for manual intervention and improves the efficiency and consistency of question generation. It not only overcomes the limitations of manual writing and simple rule-based generation methods in the prior art, but also makes full use of the advantages of natural language processing technology, providing an efficient and reliable cloze question generation method, which provides strong support for junior high school English teaching.

[0172] Embodiment 1:

[0173] In a junior high school English class in a certain city, teachers need to prepare a set of high-quality cloze exercises for students to consolidate their vocabulary and grammar knowledge. The traditional method requires teachers to select appropriate paragraphs from textbooks and extracurricular materials, and design corresponding blocking words and options. This process is time-consuming and laborious, and the quality of the generated questions is uneven. In order to improve efficiency and question quality, the school decided to use the large-model junior high school English cloze generation method based on self-supervised learning of the present invention to automatically generate cloze questions that meet the needs of junior high school English teaching.

[0174] First, the teacher collected a large amount of unlabeled English text data, including online articles, books, news reports and conversation records, and collected the city's high school entrance examination cloze test questions in the past 10 years. After cleaning and deduplication of these data, the noise data and duplicate data were removed. Next, the teacher performed word segmentation and part-of-speech tagging on the cleaned data to generate a word segmentation text data set with part-of-speech tagging. At the same time, the vocabulary of various versions of junior high school English textbooks was collected, including unit vocabulary, class vocabulary and bold key vocabulary, and these vocabulary were classified and labeled according to units and classes to form a vocabulary list for questions. Then, the pre-processed data was pre-trained with a large-scale natural language processing model using a self-supervised learning method. During the pre-training process, the model performs an occlusion prediction task, which allows the model to predict the occluded words by randomly occluding some words in the text; at the same time, it performs a contrastive learning task, which optimizes the parameters of the model by constructing positive and negative sample pairs, maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. After pre-training, the model was fine-tuned using the training dataset and text-adapted learning dataset to generate cloze questions that meet the needs of junior high school English teaching. During the fine-tuning process, the model was further optimized through supervised learning methods to ensure that the generated questions were of high quality and consistency.

[0175] Based on the trained large-scale natural language processing model and the adapted text, the cloze test questions for junior high school English are automatically generated. The model selects appropriate text paragraphs as the basic material, and randomly selects several words in the text based on the question vocabulary to block, and generates cloze test questions with options.

[0176] The following is a cloze test question generated by the method of the present invention:

[0177] Text paragraph:

[0178] The weather is getting warmer. Many people like to go outside to enjoy the sunshine. However, it is important to protect your skin from the sun. You should wear sunscreen and a hat when you go out. Drinking plenty of water is also necessary to stay hydrated.

[0179] Cloze test questions:

[0180] The weather is getting____.Many people like to go outside to enjoy the____.However, it is important to protect your skin from the____.You should wear____and a hat when you go out.Drinking plenty of____is also necessary to stay____.

[0181] Options:

[0182] 1. Warmer / hotter / cooler / dryer

[0183] 2.sunshine / rain / snow / wind

[0184] 3.sun / rain / cold / wind

[0185] 4.sunscreen / jacket / gloves / shoes

[0186] 5.water / juice / soda / milk

[0187] 6.hydrated / warm / cool / dry

[0188] In order to verify the effectiveness of the method of the present invention, we conducted a practical application test in a middle school in a certain city. In the test, we compared the efficiency and quality of the method of the present invention and the traditional method in generating cloze questions. The test time was from September 2023 to December 2023, and the test location was the second grade of a middle school in a certain city. The number of students participating in the test was 200 and the number of teachers was 10. The comparison data is shown in Table 1 below:

[0189] Table 1 Efficiency comparison data

[0190]

[0191] In the efficiency comparison with reference to Table 1, it only took 10 hours to generate 1000 cloze questions using the method of the present invention, with an average time consumption of 0.6 minutes per question, while it took 50 hours to generate 100 questions using the traditional method, with an average time consumption of 30 minutes per question. The efficiency of the method of the present invention is significantly improved.

[0192] Through the above data and comparison results, it can be seen that the method of the present invention is significantly superior to the traditional method in terms of generation efficiency and question quality, effectively solves the problems in the prior art, and provides an efficient and reliable cloze question generation solution for junior high school English teaching.

[0193] The present invention uses self-supervised learning and large-scale natural language processing models to efficiently generate a large number of cloze questions, greatly saving manpower and time costs. When generating questions, a pre-processed large-scale data set and a fine-tuned model are used to generate questions that meet the needs of junior high school English teaching. The combination of occlusion prediction tasks and contrastive learning tasks makes the generated questions more accurate in grammar and semantics, and the quality and consistency are effectively guaranteed.

[0194] The present invention realizes personalized teaching content push by dynamically adjusting and distributing the generated questions according to the students' learning data and performance. The model can automatically generate questions suitable for their learning stage according to the learning level and needs of different students, providing intelligent educational support.

[0195] The method of the present invention is applicable to various junior high school English textbook versions. By collecting and organizing the vocabulary in the textbooks, a vocabulary list for questions is generated to ensure that the generated questions are closely integrated with the teaching content. At the same time, based on the pre-training and fine-tuning of a large-scale natural language processing model, the method has high flexibility and can adapt to the teaching needs of different regions and different textbook versions.

[0196] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A large-model junior high school English cloze generation method based on self-supervised learning, characterized in that: The following steps are involved: S1. Collect a large amount of unlabeled English text data as training data set and cloze test questions from various provinces and cities in the past 10 years as text adaptation learning data set; S2. Collect and organize the vocabulary of various junior high school English textbooks and arrange them according to the parts of speech and meanings of the unit lessons and the key words in bold to form a vocabulary list for questions; S3, and preprocessing the training data set, wherein the preprocessing includes removing noise data, segmenting words, and marking parts of speech; S4. Use a self-supervised learning method in combination with a preprocessed training data set to build a large-scale natural language processing model, and pre-train the large-scale natural language processing model, wherein the self-supervised learning method includes an occlusion prediction task and a contrastive learning task, wherein the occlusion prediction task includes randomly occluding some words in English text data, and allowing the large-scale natural language processing model to predict the occluded words, and the contrastive learning task includes constructing positive and negative sample pairs, and training by maximizing the similarity of the positive sample pairs and minimizing the similarity of the negative sample pairs; S5. Fine-tune the large-scale natural language processing model according to the training data set and the text adaptation learning data set. The task fine-tuning includes applying the pre-trained large-scale natural language processing model to the adaptation of English cloze text materials and the generation of questions, and further optimizing the large-scale natural language processing model through a supervised learning method so that it can generate cloze questions that meet the needs of junior high school English teaching; S6. Automatically generate junior high school English cloze questions based on the trained large-scale natural language processing model and the adapted text, including selecting appropriate text paragraphs as basic materials, and randomly selecting a number of words in the text based on the question vocabulary list for masking, to generate cloze questions with options; S7, automatically checking and manually reviewing the generated cloze questions to ensure that the difficulty and content of the questions are suitable for the learning needs of junior high school students, wherein the automatic checking includes checking the grammatical correctness and semantic rationality of the generated questions, and the manual review includes evaluating and modifying the questions by English teaching experts; S8, storing the generated cloze questions in a question bank, and dynamically adjusting and distributing the questions according to the learning data and performance of the junior high school students; The S4 comprises the following steps: S41. Use the preprocessed English text data Dataset train Constructing a large-scale natural language processing model, wherein the large-scale natural language processing model includes an input layer, an encoding layer, a self-supervised learning task layer, and an output layer; S42. In the self-supervised learning task layer, an occlusion prediction task is designed to randomly occlude some words in the input text data and let the model predict the occluded words. The loss function of the occlusion prediction task is defined as: in, is the loss function of the occlusion prediction task, N is the number of occluded words, P(w i |w \i ) means that in the given context w \i Predict the occluded word w under the condition i The probability of S43. Design a contrastive learning task, construct positive and negative sample pairs, and train by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The loss function of the contrastive learning task is defined as: in, is the loss function of the contrastive learning task, N is the number of positive sample pairs, M is the number of negative sample pairs, and h i represents the vector of the i-th sample, Represents the vector of the i-th positive sample pair, the positive sample pair vector is the same as h i The feature vectors of similar sample pairs, Represents the vector of the jth negative sample pair, the negative sample pair vector is the same as h i The feature vectors of dissimilar sample pairs, Represents sample h i and positive sample pairs The similarity between Represents sample h i And negative sample pairs The similarity between S44. Design a sentence order prediction task to let the large-scale natural language processing model determine whether two sentences are arranged in the order of the original text. The loss function of the sentence order prediction task is defined as: in, is the loss function for the task of predicting the order of sentences, N is the number of sentence pairs, yi represents the label of whether the i-th sentence pair is in the correct order, P(y i |s i ,s i+1 ) represents the probability of the model judging the order of the i-th sentence pair; S45. Pre-train a large-scale natural language processing model by combining the occlusion prediction task, contrastive learning task and sentence order prediction task. The total loss function of the pre-training is defined as: in, is the total loss function, λ1, λ2, λ3 are adaptive weight parameters used to dynamically adjust the impact of each task loss; The S5 comprises the following steps: S51. According to the training data set Dataset train and text adaptation learning dataset adapt Fine-tuning the pre-trained large-scale natural language processing model, wherein the task fine-tuning includes applying the pre-trained model to the task of adapting and generating text materials for English cloze questions; S52, training data set Dataset train Each sentence in s i and the corresponding annotated part-of-speech vocabulary as input to generate a preliminary cloze test question: Q i ={(s i ,{(w i1 ,t i1 ),(w i2 ,t i2 ),...,(w in ,t in )})}; Among them, Q i represents the i-th generated preliminary cloze question, s i For a sentence, {(w i1 , t i1 ), (w i2 , t i2 ),...,(w in , t in )} is a vocabulary and part-of-speech tag pair; S53. Adapt the preliminary cloze test questions and combine them with the text to adapt the learning dataset adapt The passage in the middle school entrance examination cloze test real test text {q i , A i }, adjust the difficulty and content of the question, and generate the final cloze question: Q final ={(q i ,{A i },{(w i1 ,t i1 ),(w i2 ,t i2 ),...,(w in ,t in )})}; Among them, Q final represents the final generated cloze question, q i For the real test text paragraph, A i is the corresponding correct option; S54. Further optimize the large-scale natural language processing model through supervised learning methods, use the training dataset and the adapted cloze question dataset for fine-tuning training, and optimize the model parameters to minimize the loss between the generated questions and the real questions: in, is the loss function of fine-tuning training, N is the number of training samples, Q final (i) is the i-th generated cloze question, is the i-th real question; Enable the fine-tuned large-scale natural language processing model to generate cloze questions that meet the needs of junior high school English teaching; The S6 comprises the following steps: S61. Select appropriate text paragraphs as basic materials based on the trained large-scale natural language processing model, where the text paragraphs come from the preprocessed English text data Dataset train and text adaptation learning dataset adapt , the formula for selecting the appropriate text paragraph is: Among them, score(d i ) indicates paragraph d i The score of ,λ is the selection threshold, length(d i ) represents a text paragraph d i The length of freq(l j ) indicates the length l j The frequency distribution of , k is the number of candidate lengths; S62, the selected text paragraph is segmented into several sentences. The sentence segmentation formula is: Among them, split(d i ) means to add the text paragraph d i Split into a collection of sentences; S63, based on the vocabulary list V final In each sentence, a number of words are randomly selected for masking to generate cloze questions. The masking selection formula is: Q i ={(s i ,{w ij |w ij ∈V final and random()<p})|s i ∈{s1,s2,...,s n }}; Among them, Qi represents the generated cloze question, w ij is the occluded word, random() is the random function, and p is the occlusion probability; S64. Generate several options for each blocked word, including a correct option and several interference options, where the interference options are selected from words with the same part of speech and similar meanings: A ij ={w jz ,{w ijm |w ijm ∈V final and POS(w ijm )=POS(w ij )and sim(w ij ,w ijm )≥δ}}; Among them, A ij represents a set of options, w jz is the correct option, w ijm , POS(w ijm ) is the part-of-speech tag, sim(w ij , w ijm ) is the similarity function, δ is the similarity threshold; S65. Combine the generated sentences, the blocked words and their options to form the final cloze test question: Q final ={(s i ,{(w ij ,A ij )})|(s i ,{w ij })∈Q i }; Among them, Q final Represents the final generated cloze test question, including the sentence, the masked words and their corresponding option sets.

2. According to the method for generating a large-model junior high school English cloze test based on self-supervised learning according to claim 1, it is characterized in that: The S1 comprises the following steps: S11, collecting a large amount of unlabeled English text data, where the sources of the English text data include online articles, books, news reports and conversation records; S12. Clean and deduplicate the collected English text data to remove noise data and duplicate data: D clean ={d i ∈D|if score(d i )≥λ}; Where D represents the initial data set, d i represents the i-th data in the data set, score(d i ) represents data d i The noise score, λ is the threshold for removing noise, D clean represents the cleaned data set; S13. Perform word segmentation on the cleaned English text data, split each sentence into word sequences, and perform part-of-speech tagging on each word to generate a word segmentation text dataset with part-of-speech tagging: {(w1,t1),(w2,t2),...,(w n ,t n )}=POS_Tagger(Tokenizer(D clean )); Among them, w i represents the i-th word, t i Represents the part-of-speech tag of the i-th word, Tokenizer is the word segmentation algorithm, and POS_Tagger is the part-of-speech tagging algorithm; S14. Collect the cloze test papers of the senior high school entrance examinations in various provinces and cities in the past 10 years, including the cloze test papers of the senior high school entrance examinations in previous years; S15. Sort and annotate the collected cloze test questions, match the text paragraph of each question with the corresponding correct option, and mark the test text as a text adaptation learning dataset: Q prep ={(q i ,A i )|q i ∈Q,A i =Match(q i )}; Among them, Q represents the collection of real questions collected, q i represents the i-th question, Ai represents the correct option for the i-th question, Match is the matching function, Q prep Represents the sorted real-question dataset; S16, construct a training data set and a text adaptation learning data set, and save the part-of-speech annotated English text data after word segmentation and the high school entrance examination cloze test text adaptation learning data set separately: Dataset train ={{(w1,t1),(w2,t2),...,(w n ,t n )}}; Dataset adapt =Q prep ; Among them, Dataset train Represents the training data set after word segmentation and part-of-speech tagging, Dataset adapt Represents the sorted real test text adaptation learning dataset.

3. According to the method for generating a large-scale junior middle school English cloze test based on self-supervised learning according to claim 2, it is characterized in that: The S2 comprises the following steps: S21, collecting vocabulary from various versions of English textbooks for junior high schools, including unit vocabulary, class vocabulary and boldface key vocabulary in the textbooks, to form a preliminary vocabulary list; S22, sorting the preliminary vocabulary list, classifying the vocabulary according to units and classes, and marking the part of speech and meaning of each vocabulary word to generate a classified and annotated vocabulary list: V unit ={(w i ,p i ,m i )|w i ∈W,p i =POS(w i ),m i =Meaning(w i )}; Among them, V unit represents a vocabulary classified by unit, w i represents the i-th word, p i Represents vocabulary w i Part of speech, m i Represents vocabulary w i POS is the part-of-speech tagging function, and Meaning is the meaning tagging function; S23, extracting the boldface key words from the classification and annotation vocabulary list to form a boldface key words list: V bold ={w i ∈V unit |if isBold(w i )}; Among them, V bold represents the bold emphasis vocabulary list, isBold is a function to determine whether a word is a bold emphasis word; S24. Sort and arrange the classified and annotated vocabulary lists and the boldface key vocabulary lists according to units and lessons to form the final vocabulary list for questions: V final ={(w i ,p i ,m i )∈V unit ∪V bold }; Among them, V final Indicates the final vocabulary list for the test, including classified and marked vocabulary and bold key vocabulary.

Images