A method for judging the difficulty of English text based on Llama3

Abstract

The present invention relates to a method for judging the difficulty of an English text based on Llama3, and belongs to the relevant fields of deep learning, natural language processing, large models, natural language generation, etc. First, the features required for the difficulty of an English text are extracted by designing a special prompt word, and then the extracted features are combined with the original English text to fill in the prompt word template used for training to construct training data, and a loss function is designed, and finally, the text difficulty label is used as a supervisory signal to adjust the Llama3 model, and the adjusted model is used for difficulty judgment to realize English text difficulty judgment. The present invention can realize English text difficulty judgment, and compared with the existing method, the present method can effectively improve the accuracy of difficulty judgment.

Description

Technical Field

[0001] The present invention relates to a method for judging the difficulty of an English text based on Llama3, and belongs to relevant fields such as deep learning, natural language processing, large models, and natural language generation. Background Art

[0002] With the growing demand for personalized learning, the difficulty judgment of English texts has become one of the factors affecting personalized education and adaptive learning systems. The difficulty judgment of English texts is crucial for both learners and educators. It can help learners choose the text materials that best suit their current English level, improve learning efficiency and gain a good learning experience. It can also help educators formulate teaching plans more effectively, ensuring that students will not feel bored because the text is too simple, nor frustrated because it is too difficult. Therefore, using advanced technical means to judge the difficulty of texts has become an effective way to improve the accuracy of English text difficulty judgment.

[0003] Since the beginning of the 20th century, the academic community has begun research on the difficulty of English texts, including using lexical analysis to identify rare words and advanced vocabulary, analyzing the complexity of sentence structure, calculating readability scores through readability formulas (such as Flesch-Kincaid), and evaluating the subject complexity and required background knowledge of the text. In addition, the impact of contextual and cultural factors on text comprehension is also considered, and a comprehensive evaluation is conducted in combination with the feedback of target readers. However, with the advent of the information age, text content has become more complex, and traditional rule-based methods are often too simple and lack sufficient generalization to cope with this complexity, resulting in poor results and a lack of immediacy.

[0004] The emergence of large language models (LLMs) such as ChatGPT and Llama3 has brought a turning point for the judgment of English text difficulty. As the latest open source large language model released by Meta, the Llama3 model not only excels in multilingual support and long text processing, but also in common sense reasoning, text simplification and other aspects. Llama3 has significant advantages in the field of English text difficulty judgment, which is mainly due to its two core features: sufficient pre-training and instruction adjustment technology. This technology allows the model to be optimized through task-specific data while maintaining the original pre-training knowledge, so that the model can more accurately judge texts of different difficulty levels. In addition, in the adjustment process of Llama3, PEFT (Parameter-Efficient Fine-Tuning) efficient parameter adjustment methods including Prefix Tuning (adding trainable prefix embeddings to the model input layer), LoRA (updating model parameters through low-rank matrix approximation), and Adapter Tuning (inserting small neural network adapters between model layers) can be used. These methods reduce memory usage and computational costs by updating only a small part of the model parameters, while maintaining or even improving the performance of the model. In this way, Llama3 can more flexibly adapt to education and other fields that require text difficulty judgment, providing students with customized reading experiences and learning resources. The adjusted Llama3 can not only understand the surface meaning of the text, but also deeply analyze the text structure, vocabulary complexity and concept difficulty, providing an efficient and reliable solution for judging the difficulty of English texts. Accurate judgment of the difficulty of English reading materials is of great significance to promoting personalized learning. Using Llama3 adjustment technology can effectively improve the accuracy and efficiency of this judgment. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for judging the difficulty of English text based on Llama3, and to solve the problem that the traditional method has inaccurate difficulty judgment of English text.

[0006] The technical solution of the present invention is: a method for judging the difficulty of English text based on Llama3, and the specific steps are: Step 1: establishing a corpus to extract the features required for predicting the difficulty of text; Step 2: constructing training data and loss function according to the extracted features; Step 3: using the training data and loss function to complete model training and realize English text difficulty judgment.

[0007] Step 1.1: Set up a corpus , including N English texts, and , represents the difficulty label of N English texts. The difficulty judgment is divided into two levels: simple and difficult. Each difficulty label contains 0 or 1, where 0 represents a simple text and 1 represents a difficult text. The difficulty label is related to the training data set. Taking the College English Test Band 4 and Band 6 as an example, the difficulty label of the Band 4 test text is 0, and the difficulty label of the Band 6 test text is 1;

[0008] Step 1.2: Set the prompt word template for difficult vocabulary extraction as , extract difficult vocabulary features, prompt word template Specifically:

[0009] "This is an English text, the specific content is: ;

[0010] Please find the words in the text that may increase the reading difficulty and separate them with commas";

[0011] in, Indicates that you fill in the text here. Fill in the template , and input it into a large language model to identify difficult words and obtain M difficult words ;

[0012] Step 1.3: Set the prompt word template for calculating the depth of the syntax tree (the depth of the syntax tree refers to the number of nodes passed by the longest path from the root node to the deepest leaf node in the syntax tree. The depth of the syntax tree measures the nesting level or complexity of the grammatical structure. For example, in the sentence "The quick brown fox jumps over the lazy dog.", the deepest path of this syntax tree is S → VP → PP → NP → dog, that is, the depth is 5) is , extract syntax tree deep features, prompt word template Specifically:

[0013] "This is an English text, the specific content is: ;

[0014] Please directly return the syntax tree depth of the sentence";

[0015] in, Indicates that you fill in the text here. Fill in the prompt word template , and input it into a large language model to obtain the syntax tree depth .

[0016] Step 2. Construct training data and loss function:

[0017] Step 2.1: Set up the prompt word template for model adjustment , prompt word template Specifically:

[0018] "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.### Instruction:Please judge the difficulty of this sentence###Input:Difficulty words: ;syntax tree depth:{T};sentence:{S}### Response: "

[0019] in, Indicates to fill in the difficult vocabulary here. Indicates that the syntax tree depth is filled in here. Indicates that the text to be judged for difficulty is filled in here;

[0020] Step 2.2: The corpus and the difficulty vocabulary of each sentence obtained And the syntax tree depth , enter the prompt word template in sequence , get the training corpus , tagged as , complete the training data construction;

[0021] Step 2.3: Set the loss function. In addition to the autoregressive loss used by the Llama3 model, add the difficulty judgment task loss. Suppose the training corpus The output after inputting the Llama3 model is If the Llama3 model has not yet learned the output format at the beginning of training, If it is neither 0 nor 1, then set is 1 (the same applies to 0), as the adjustment proceeds, no If it is neither 0 nor 1, the difficulty judgment task loss function is:

[0022]

[0023] The difficulty judgment task loss represents the deviation between the predicted result and the true label. As the training progresses, the difficulty judgment task loss continues to decrease, and finally the Llama3 model output result fits the difficulty label. The total loss used in the training is the sum of the original loss used by the Llama3 model and the difficulty judgment task loss, that is:

[0024]

[0025] in, Represents the original loss used by the Llama3 model.

[0026] Step 3. Train the model and generate results:

[0027] Step 3.1: After loading the initial Llama3 training model using the LoRA method, load the training data, set the loss function, and finally complete the Llama3 model adjustment;

[0028] Step 3.2: Use the adjusted Llama3 model to judge the difficulty, and input a text to be judged into the prompt word template and the prompt word template In the example, the difficult vocabulary features and the syntax tree depth features are obtained respectively, and then the text, the difficult vocabulary features and the syntax tree depth features are filled into the prompt word template. After getting the complete prompt word, input it into the adjusted Llama3 model to get the difficulty judgment result. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Example 1: Figure 1 As shown, a teaching plan generation method for high school English reading class, the specific steps are:

[0032] Step 1: Establish a corpus and extract the features needed to predict the difficulty of text; Step 2: Construct training data and loss function based on the extracted features; Step 3: Use training data and loss function to complete model training and realize English text difficulty judgment.

[0033] Step 1.1: Set up a corpus , including N English texts, and , represents the difficulty label of N English texts. The difficulty judgment is divided into two levels: simple and difficult. Each difficulty label contains 0 or 1, where 0 represents a simple text and 1 represents a difficult text. The difficulty label is related to the training data set. Taking the College English Test Band 4 and Band 6 as an example, the difficulty label of the Band 4 test text is 0, and the difficulty label of the Band 6 test text is 1;

[0034] Step 1.2: Set the prompt word template for difficult vocabulary extraction as , extract difficult vocabulary features, prompt word template Specifically:

[0035] "This is an English text, the specific content is: ;

[0036] Please find the words in the text that may increase the reading difficulty and separate them with commas";

[0037] in, Indicates that you fill in the text here. Fill in the prompt word template , and input it into a large language model to identify difficult words and obtain M difficult words , if the input text is "Despite the seemingly insurmountable obstacles that arose at every turn, she remained steadfast in her pursuit of the truth, meticulously dissecting each piece of evidence to uncover the hidden layers of deception.", then the prompt words constructed after filling in the template are:

[0038] "This is an English text. The specific content is: "Despite the seemingly insurmountable obstacles that arose at every turn, she remained steadfast in her pursuit of the truth, meticulously dissecting each piece of evidence to uncover the hidden layers of deception."

[0039] Please find the words in the text that may increase the reading difficulty and separate them with commas";

[0040] The results are [seemingly, insurmountable, obstacles, arose, steadfast];

[0041] Step 1.3: Set the prompt word template for calculating the depth of the syntax tree (the depth of the syntax tree refers to the number of nodes passed by the longest path from the root node to the deepest leaf node in the syntax tree. The depth of the syntax tree measures the nesting level or complexity of the grammatical structure. For example, in the sentence "The quick brown fox jumps over the lazy dog.", the deepest path of this syntax tree is S → VP → PP → NP → dog, that is, the depth is 5) is , extract syntax tree deep features, prompt word template Specifically:

[0042] "This is an English text, the specific content is: ;

[0043] Please directly return the syntax tree depth of the sentence";

[0044] in, Indicates that you fill in the text here. Fill in the prompt word template , and input it into a large language model to obtain the syntax tree depth ,Take the above specific sentence as an example, the prompt words constructed after filling in the template are:

[0045] "This is an English text. The specific content is: "Despite the seemingly insurmountable obstacles that arose at every turn, she remained steadfast in her pursuit of the truth, meticulously dissecting each piece of evidence to uncover the hidden layers of deception."

[0046] Please directly return the syntax tree depth of the sentence"

[0047] The output result of the large model is 6.

[0048] Step 2, construct training data and loss function:

[0049] Step 2.1: Set up the prompt word template for model adjustment , prompt word template Specifically:

[0050] "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.### Instruction:Please judge the difficulty of this sentence###Input:Difficulty words: ;syntax tree depth:{T};sentence:{S}### Response: "

[0051] in, Indicates to fill in the difficult vocabulary here. Indicates that the syntax tree depth is filled in here. Indicates to fill in the text to judge the difficulty here, the specific filling process is the same as above;

[0052] Step 2.2: The corpus and the difficulty vocabulary of each sentence obtained And the syntax tree depth , enter the prompt word template in sequence , get the training corpus , tagged as , complete the training data construction;

[0053] Step 2.3: Set the loss function. In addition to the autoregressive loss used by the Llama3 model, add the difficulty judgment task loss. Suppose the training corpus The output after inputting the Llama3 model is If the Llama3 model has not yet learned the output format at the beginning of training, If it is neither 0 nor 1, then set is 1 (the same applies to 0), as the adjustment proceeds, no If it is neither 0 nor 1, the difficulty judgment task loss function is:

[0054]

[0055] The difficulty judgment task loss represents the deviation between the predicted result and the true label. As the training progresses, the difficulty judgment task loss continues to decrease, and finally the Llama3 model output result fits the difficulty label. The total loss used in the training is the sum of the original loss used by the Llama3 model and the difficulty judgment task loss, that is:

[0056]

[0057] in, Represents the original loss used by the Llama3 model.

[0058] Step 3. Train the model and generate results:

[0059] Step 3.1: After loading the initial Llama3 training model using the LoRA method, load the training data, set the loss function, and finally complete the Llama3 model adjustment;

[0060] Step 3.2: Use the adjusted Llama3 model to judge the difficulty, and input a text to be judged into the prompt word template and the prompt word template In the example, the difficult vocabulary features and the syntax tree depth features are obtained respectively, and then the text, the difficult vocabulary features and the syntax tree depth features are filled into the prompt word template. After getting the complete prompt word, input it into the adjusted Llama3 model to get the difficulty judgment result.

[0061] In order to verify the effect of the present invention, experiments were conducted on two NVIDIA A100 Tensor Core GPUs using unsloth 2024.8, transformers 4.43, and xformers 0.0.25. The public datasets CEFR and Newsela were used to compare the performance by accuracy (ACC), area under the curve (AUC), and root mean square error (RMSE). The ChatGPT model was called using the gpt-3.5-turbo-instruct interface, and the model was adjusted to Epoch 15. Experiments were conducted under a variety of hyperparameters, and the best parameters in the validation set were selected to test the test set. The experimental conditions were consistent with those in the literature [1], and the experimental results of the comparison model were all from the literature [1]. The experimental results are shown in Table 1.

[0062] Table 1 Experimental results of different models on CEFR and Newsela datasets

[0063]

[0064] As can be seen from Table 1, the AUC, ACC and RMSE of the present invention on the CEFR dataset are 0.9506, 0.9512 and 0.2208 respectively, and the AUC, ACC and RMSE on the Newsela dataset are 0.9153, 0.8398 and 0.3312 respectively, which are better than other baseline models, thus proving the effectiveness of the present invention.

[0065] [1]Wang Y, Zhou J, Li Z, et al. Automatically difficulty gradingmethod for English reading corpus with Multi-feature Embedding based on Pre-trained Language Model[J]. IEEE Transactions on Learning Technologies, 2023.

[0066] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for judging the difficulty of English text based on Llama3, characterized by: Step 1: Build a corpus and extract the features needed to predict text difficulty; Step 2: Construct training data and loss function based on the extracted features; Step 3: Use the training data and loss function to complete the Llama3 model training and realize the difficulty judgment of English text; The Step 1 is specifically as follows: Step 1.1: Set up a corpus , including N English texts, and , represents the difficulty label of N English texts. The difficulty judgment is divided into two levels: easy and difficult. Each difficulty label contains 0 or 1, 0 represents easy text and 1 represents difficult text; Step 1.2: Set up a prompt word template for difficult vocabulary extraction , extract difficult vocabulary features, prompt word template Specifically: "This is an English text, the specific content is: ; Please find the words in the text that may increase the reading difficulty and separate them with commas"; in, Indicates that you fill in the text here. Fill in the prompt word template , and input it into a large language model to identify difficult words and obtain M difficult words ; Step 1.3: Set the prompt word template for calculating the depth of the syntax tree , extract syntax tree deep features, prompt word template Specifically: "This is an English text, the specific content is: ; Please return the syntax tree depth of the sentence directly"; in, Indicates that you fill in the text here. Fill in the prompt word template , and input it into a large language model to obtain the syntax tree depth ; The syntax tree depth is used to measure the nesting level or complexity of the syntax structure; The Step 2 is specifically as follows: Step 2.1: Set the prompt word template for model adjustment , prompt word template Specifically: "Below is an instruction that describes a task, paired with an input thatprovides further context. Write a response that appropriately completes therequest.### Instruction:Please judge the difficulty of this sentence;###Input:Difficulty words: ;syntax tree depth:{T};sentence:{S}### Response: " in, Indicates to fill in the difficult vocabulary here. Indicates that the obtained syntax tree depth is filled in here. Indicates that the text to be judged for difficulty is filled in here; Step 2.2: The corpus and difficult vocabulary And the syntax tree depth , enter the prompt word template in sequence , get the training corpus , tagged as , complete the training data construction; Step 2.3: Set the loss function. In addition to the autoregressive loss used by the Llama3 model, add the difficulty judgment task loss. Suppose the training corpus The output after inputting the Llama3 model is If the Llama3 model has not yet learned the output format at the beginning of training, If it is neither 0 nor 1, then set If is 1, the difficulty judgment task loss function is: ； The difficulty judgment task loss represents the deviation between the predicted result and the true label. As the training progresses, the difficulty judgment task loss continues to decrease, and finally the Llama3 model output result fits the difficulty label. The total loss used in the training is the sum of the original loss used by the Llama3 model and the difficulty judgment task loss, that is: ； in, Represents the original loss used by the Llama3 model.

2. The method for determining the difficulty of an English text based on Llama3 according to claim 1, characterized in that: The Step 3 is specifically as follows: Step 3.1: After loading the initial Llama3 training model using the LoRA method, load the training data, set the loss function, and finally complete the Llama3 model adjustment; Step 3.2: Use the adjusted Llama3 model to judge the difficulty, and input a text to be judged into the prompt word template and the prompt word template In the example, the difficult vocabulary features and the syntax tree depth features are obtained respectively, and then the text, the difficult vocabulary features and the syntax tree depth features are filled into the prompt word template. After getting the complete prompt word, input it into the adjusted Llama3 model to get the difficulty judgment result.

Images