A controllable text simplification method for graded reading in English

Abstract

The present invention relates to a controllable text simplification method for graded reading of English subjects, and belongs to the technical field of text simplification. First, a special data set in the form of [complex sentence, simple sentence, control token] is constructed; secondly, an input template for explicit controllability is constructed, and the special data set is integrated to obtain the input of the Llama3.1 model; then, a loss function for simplified generation and controllable token regression is constructed to obtain a special trainer; the input, output results and special trainer of the Llama3.1 model are used to adjust the Llama3.1 model based on the LoRA method; finally, a model input instruction is constructed, and a simple sentence is output through the adjusted Llama3.1 model. The present invention can realize the English text simplification function with explicit indicator control, input the explicit control token into the model, and achieve the effect of supervised training to a certain extent without changing the model structure, effectively improving the controllability and simplification performance of the text simplification algorithm.

Description

Technical Field

[0001] The invention relates to a controllable text simplification method for graded reading of English subjects, belonging to the technical field of text simplification. Background Art

[0002] The Llama series of models are a series of open source large-scale language models developed by Meta. These models are designed to provide efficient text generation and understanding capabilities while reducing the computational cost during training and inference. The Llama series provides models with a variety of parameter sizes and optimizes the architecture and training process, reducing the required computing resources, and performs well on multiple benchmark tasks, especially generative tasks. Its excellent performance is inseparable from the Decoder-only architecture adopted by the Llama series of models. This structure only uses the decoder part of the Transformer, making the architecture more concise and more suitable for text generation tasks. At the same time, the optimized Decoder-only model can use parallel computing during training, greatly improving efficiency. The latest Llama3.1 has significantly improved the context window, the number of supported languages, and the number of tokens compared to the Llama2 model. Its 8B and 70B versions both surpass other open source models of the same size and perform well in various benchmarks.

[0003] In deep learning, a trainer is a high-level tool or module used to manage and execute the training process. It encapsulates the core steps of model training, making the entire training process more automated, modular, and easy to manage. The trainer selected in this method is the Trainer class in the Transformers library, which is a high-level API provided by Hugging Face to simplify the process of training and evaluating transformer models. The Trainer class encapsulates multiple complex steps such as model training, evaluation, logging, and model saving, helping users to quickly implement model training and deployment. It also provides a flexible API for advanced users to customize, greatly simplifying the use process of transformer models. The core part of it is the compute_loss method, which is responsible for calculating the loss function that needs to be minimized during the training process. If the user wants to customize the loss function, he can rewrite the compute_loss method to return the calculated loss value, so as to update the trainable parameters of the model according to the requirements of the specific task in the backpropagation stage.

[0004] LoRA is a lightweight model adjustment method specifically designed for efficient adaptation and adjustment of large language models (such as GPT, BERT, etc.). By adding a small-scale low-rank matrix to the original parameter matrix and freezing the original parameter matrix during the adjustment process and only updating the parameters of the low-rank matrix, the computational and storage costs of large model adjustment are effectively reduced. It is an efficient and flexible model adaptation method.

[0005] Control tokens play a key role in controllable text generation. They are used to guide the generative model to output text that meets specific requirements. Control tokens can be directly embedded into the input text as part of the generation conditions, explicitly telling the model what type of text needs to be generated. Users can flexibly control specific aspects of the generated content without retraining the model.

[0006] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a controllable text simplification method for graded reading of English subjects, which solves the problem that traditional text simplification methods are insufficient in controllability and simplification performance without changing the model structure.

[0008] The technical solution of the present invention is: a controllable text simplification method for graded reading of English subjects, the specific steps are:

[0009] Step 1: Construct a dedicated dataset in the form of [complex sentences, simple sentences, control tokens];

[0010] For a given complex sentence There are various simplified versions of the original data ,in For complex sentences Different artificial simplification results. and simplified sentences As input, the control token required is ,get and The ratio between the number of words, the number of characters, the character similarity, the word difficulty, and the grammatical similarity. The formula is as follows:

[0011]

[0012] Obtain the control token of each artificially simplified sentence according to the above method, and construct each simple sentence, complex sentence and its control token into a dedicated data set

[0013] Step 2: Build an input template for explicit controllability and integrate a dedicated dataset to obtain the input of the Llama3.1 model;

[0014] Step 2.1 builds a prompt template that can integrate all control tokens to explicitly simplify complex sentences in a controllable manner, as follows:

[0015] Prompt: "Below is an instruction that describes a task, paired with aninput that provides further context. Write a response that appropriatelycompletes the request.### Instruction:Simplify the sentence given below by referring to the control tokens.### Input:W_{W} C_{C} L_{L} WR_{WR} DTD_{DTD}{comp}### Response: "

[0016] Since the Llama 3.1 model only supports English input, the input template for explicit controllability described in the present invention can only be presented in English, which means: "The following is an instruction describing a task, paired with input that provides further context. Write a response that properly completes the request. Instructions: Simplify the sentence given below with reference to control tokens. Input: {control token} {complex sentence}. Output: {simple sentence}."

[0017] Step 2.2: The control token and complex sentences Integrate into the Prompt template to form Llama3.1 model input ,by As a label ,Right now:

[0018]

[0019]

[0020] The resulting model inputs and labels are as follows:

[0021] Example Input:"Below is an instruction that describes a task, paired with an input that provides further context. Write a response thatappropriately completes the request.### Instruction:Simplify the sentencegiven below by referring to the control tokens.### Input:W_{ } C_{ } L_{ } WR_{ } DTD_{ } { }### Response: "

[0022] Example Label: .

[0023] Step 3: Construct a loss function for simplified generation and controllable token regression to obtain a dedicated trainer.

[0024] Using a trainer inherited from the trainer class of the transformers library of HuggingFace, the calculation method of the training loss of the trainer is rewritten to obtain a dedicated trainer. During the model training process, the dedicated trainer first determines the training loss according to the new calculation method, and then uses the gradient information of the training loss to perform the back propagation algorithm to optimize and update the parameters of the model. This method enables the training process to reflect specific performance indicators. The specific performance indicator refers to the MSE (mean square error) between the control token of the generated text and the control token of the label, which reflects the controllable characteristics of the generated text. The trainer loss calculation method is to calculate the standard loss of the simplified text generated by the model and the custom control token loss, that is:

[0025]

[0026] in is the standard loss for the model to generate simplified text, which is the target simple sentence (label ) and the simple sentences generated by the model The cross entropy loss between is:

[0027]

[0028] is the loss of the introduced control token, which calculates the generation of the target simple sentence (label ) control token And the simple sentence generated by the model Control Token The mean absolute error between them is:

[0029]

[0030] in, is the proportionality coefficient.

[0031] Step 4: Use the input and output results of the Llama3.1 model and the dedicated trainer to adjust the Llama3.1 model based on the LoRA method;

[0032] Based on the unsloth framework, load the Llama3.1 model and the corresponding LoRA adapter, and apply LoRA to the model. Use the dedicated trainer to load the dedicated dataset and input the Llama3.1 model with the Prompt model. The trainer can back-propagate and update the parameters of the low-rank matrix introduced by the LoRA method according to the configured information after each training loss is calculated according to the model loss function. After the training is completed, save the adjusted model weights and call the adjusted model in subsequent tasks.

[0033] Step 5: Build the model input command and output a simple sentence through the adjusted Llama3.1 model:

[0034] For a new complex sentence , the model generates simple sentences The control token is set to , put the new complex sentence and the control token into the Prompt template to obtain the input of the model ,Right now:

[0035]

[0036] Will Input the adjusted Llama3.1 model, use the model generation method (model.generate) to get the token sequence output by the model, and then decode this token sequence using the tokenizer of the Llama3.1 model to get the output simple sentence :

[0037] =Fine-tuned Llama3.1

[0038] Among them, Fine-tuned Llama3.1 Indicates that Input the adjusted Llama3.1 model to get the output .

[0039] The beneficial effects of the present invention are as follows: compared with the prior art, the present invention can realize the English text simplification function with explicit indicator control, input explicit control tokens into the model, and achieve the effect of supervised training to a certain extent without changing the model structure, thereby effectively improving the controllability and simplification performance of the text simplification algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0041] The present invention proposes a controllable text simplification method for graded reading of English subjects. Llama3.1-8b is used as the base model to realize the English text simplification function with explicit indicator control. The validation set part in ASSET and Turk data sets is used as all the training data of this paper. For controllable simplified text generation, explicit control tokens are input into the model. Without changing the model structure, the effect of supervised training is achieved to a certain extent, and the controllability of the text simplification algorithm is effectively improved.

[0042] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0044] Example 1: Figure 1 As shown in FIG. 1 , a controllable text simplification method for graded reading of English subjects is provided, and the specific steps are as follows:

[0045] Step 1: Construct a dedicated dataset in the form of [complex sentences, simple sentences, control tokens];

[0046] For a given complex sentence There are various simplified versions of the original data ,in For complex sentences Different artificial simplification results. and simplified sentences As input, the control token required is ,get and The ratio between the number of words, the number of characters, the character similarity, the word difficulty, and the grammatical similarity. The formula is as follows:

[0047]

[0048] Obtain the control token of each artificially simplified sentence according to the above method, and construct each simple sentence, complex sentence and its control token into a dedicated data set

[0049] Step 2: Build an input template for explicit controllability and integrate a dedicated dataset to obtain the input of the Llama3.1 model;

[0050] Step 2.1 builds a prompt template that can integrate all control tokens to explicitly simplify complex sentences in a controllable manner, as follows:

[0051] Prompt: "Below is an instruction that describes a task, paired with aninput that provides further context. Write a response that appropriatelycompletes the request.### Instruction:Simplify the sentence given below by referring to the control tokens.### Input:W_{W} C_{C} L_{L} WR_{WR} DTD_{DTD}{comp}### Response: "

[0052] Since the Llama 3.1 model only supports English input, the input template for explicit controllability described in the present invention can only be presented in English, which means: "The following is an instruction describing a task, paired with input that provides further context. Write a response that properly completes the request. Instructions: Simplify the sentence given below with reference to control tokens. Input: {control token} {complex sentence}. Output: {simple sentence}."

[0053] Step 2.2: The control token and complex sentences Integrate into the Prompt template to form Llama3.1 model input ,by As a label ,Right now:

[0054]

[0055]

[0056] The resulting model inputs and labels are as follows:

[0057] Example Input:"Below is an instruction that describes a task, paired with an input that provides further context. Write a response thatappropriately completes the request.### Instruction:Simplify the sentencegiven below by referring to the control tokens.### Input:W_{ } C_{ } L_{ } WR_{ } DTD_{ } { }### Response: "

[0058] Example Label: .

[0059] Step 3: Construct a loss function for simplified generation and controllable token regression to obtain a dedicated trainer; the loss function combines cross entropy loss and custom control token loss, where the custom loss is measured by mean square error (MSE). This loss design allows the model to optimize the generation quality while learning the requirements of control indicators when generating text, such as the character length of the text, vocabulary complexity, etc.

[0060] Using the trainer class inherited from the transformers library of HuggingFace, the calculation method of the training loss of the trainer is rewritten to obtain a dedicated trainer. During the model training process, the dedicated trainer first determines the training loss according to the new calculation method, and then uses the gradient information of the training loss to perform the back propagation algorithm to optimize and update the parameters of the model so that the training process can reflect specific performance indicators. The specific performance indicator refers to the MSE (mean square error) between the control token of the generated text and the control token of the label, which reflects the controllable characteristics of the generated text. The trainer loss calculation method is to calculate the standard loss of the simplified text generated by the model and the custom control token loss, that is:

[0061]

[0062] in is the standard loss for the model to generate simplified text, which is the target simple sentence (label ) and the simple sentences generated by the model The cross entropy loss between is:

[0063]

[0064] is the loss of the introduced control token, which calculates the generation of the target simple sentence (label ) control token And the simple sentence generated by the model Control Token The mean absolute error between them is:

[0065]

[0066] in, is the proportionality coefficient.

[0067] Step 4: Use the input and output results of the Llama3.1 model and the dedicated trainer to adjust the Llama3.1 model based on the LoRA method;

[0068] Based on the unsloth framework, load the Llama3.1 model and the corresponding LoRA adapter, and apply LoRA to the model. Use the dedicated trainer to load the dedicated data set, and input the Llama3.1 model with the Prompt model. The trainer can back-propagate and update the parameters of the low-rank matrix introduced by the LoRA method according to the configured information after each training loss is calculated according to the loss function. After the training is completed, save the adjusted model weights, and call the adjusted model in subsequent tasks.

[0069] Step 5: Build the model input command and output a simple sentence through the adjusted Llama3.1 model:

[0070] For a new complex sentence , the model generates simple sentences The control token is set to , put the new complex sentence and the control token into the Prompt template to obtain the input of the model ,Right now:

[0071]

[0072] Will Input the adjusted Llama3.1 model, use the model generation method (model.generate) to get the token sequence output by the model, and then decode this token sequence using the tokenizer of the Llama3.1 model to get the output simple sentence :

[0073] =Fine-tuned Llama3.1

[0074] Among them, Fine-tuned Llama3.1 Indicates that Input the adjusted Llama3.1 model to get the output .

[0075] In order to verify the effect of the present invention, experiments were conducted on two NVIDIA A100 Tensor Core GPUs using unsloth 2024.8, transformers 4.43., and xformers 0.0.25. The public datasets TurkCorpus and ASSET were used, and the SARI, BLEU, and FKGL indicators reported in the experiment were calculated using the EASSE tool. The model was adjusted to Epoch 10, and the ratio of simplified Loss to control token Loss was 1:10, that is, , the target control token is set to W_1.05 C_0.95 WR_0.85 L_0.85 DTD_0.85. Experiments are conducted under various hyperparameters, and the best parameters in the validation set are selected to test the test set. The experimental results are shown in Tables 1, 2, and 3.

[0076] Table 1 Controllable performance of the method and Control T5 (MAE ↓)

[0077]

[0078] Table 2 Controllability performance of Llama3.1 CTS and Control T5 (RMSE ↓)

[0079]

[0080] MAE is the average of the absolute errors of all samples. It measures the average degree of difference between the predicted value and the true value, regardless of the direction of the difference. RMSE is the square root of the average of the squared errors of all samples. Similar to MAE, RMSE also measures the difference between the predicted value and the true value, but is more sensitive to outliers (i.e., very large errors). The smaller the RMSE and MAE, the higher the prediction accuracy of the model. The data in Tables 1 and 2 show that compared with the baseline model Control-T5, the method of the present invention performs well in controllability.

[0081] Table 3: SARI↑, BLEU↑, FKGL↓ performance of the proposed method and other baseline models

[0082]

[0083] BLEU is a commonly used metric for measuring the quality of machine translation, which evaluates the accuracy of translation by analyzing the degree of vocabulary and phrase matching between the generated text and the reference text. However, the limitation of BLEU is that it fails to fully consider sentence structure and deep semantics, which may result in a lower score due to differences in word usage even if the generated text is semantically consistent with the reference text. Nevertheless, in order to compare with the performance of earlier text simplification systems, we also report the results of the BLEU metric.

[0084] SARI is the main indicator for evaluating the performance of text simplification systems. It compares the original text, the generated simplified text, and the reference simplified text to comprehensively evaluate the quality of the text in terms of retaining key information, removing redundant content, and adding necessary information. The SARI indicator provides a comprehensive evaluation framework for text simplification systems.

[0085] FKGL is an indicator of text readability. It converts the complexity of a text into a corresponding education level to indicate whether the text is suitable for readers with a specific educational background. The lower the FKGL value, the higher the readability of the text and the more suitable it is for readers with a lower education level.

[0086] The analysis results in Table 3 reveal that the method of the present invention achieves significant performance improvement in the SARI indicators of the ASSET and TurkCorpus datasets compared to all baseline models. Specifically, the method achieved excellent scores of 47.14 and 43.69 on these two datasets, respectively, and achieved significant improvements of 2.10 and 0.69 respectively compared with the Control-T5 model. In addition, through comparative analysis of the BLEU and FKGL indicators, the performance of the method on the ASSET dataset is particularly outstanding, which shows that the method makes more flexible and diverse adjustments to the sentence structure while maintaining semantic integrity. At the same time, the method adopts a more concise expression in the selection of vocabulary and phrases. Although this may lead to a lower match degree in the BLEU indicator, it shows higher performance in the SARI indicator. The data in Tables 1, 2 and 3 effectively verify the effectiveness of the method of the present invention.

[0087] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0088] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0089] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0090] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A controllable text simplification method for graded reading of English subjects, characterized by: Step 1: Build a dedicated dataset in the form of [complex sentences, simple sentences, control tokens]; Step 2: Build an input template for explicit controllability and integrate a dedicated dataset to obtain the input of the Llama3.1 model; Step 3: Construct a loss function for simplified generation and controllable token regression to obtain a dedicated trainer; Step 4: Use the input and output results of the Llama3.1 model and the dedicated trainer to adjust the Llama3.1 model based on the LoRA method; Step 5: Construct Llama3.1 model input instructions, and output simple sentences through the adjusted Llama3.1 model; The Step 1 is specifically as follows: For a given complex sentence There are various simplified versions of the original data ,in For complex sentences Different artificial simplification results of complex sentences and simplified sentences As input, the control token required is ,get and The ratio of the number of words, the number of characters, the character similarity, the word difficulty, and the grammatical similarity is as follows: ； Obtain the control token of each artificially simplified sentence according to the above method, and construct each simple sentence, complex sentence and the control token into a dedicated data set ; The construction of an explicitly controllable input template includes: Build a prompt template that can integrate all control tokens to explicitly simplify complex sentences in a controlled manner, as follows: Prompt: "Below is an instruction that describes a task, paired with aninput that provides further context. Write a response that appropriatelycompletes the request.### Instruction:Simplify the sentence given below by referring to the control tokens.### Input:W_{W} C_{C} L_{L} WR_{WR} DTD_{DTD}{comp}### Response: " Since the Llama 3.1 model only supports English input, the input template for explicit controllability can only be presented in English, which means: "Below is an instruction that describes a task. Paired with input that provides further context, write a response that appropriately completes the request. Instructions: Simplify the sentence given below with reference to control tokens. Input: {control tokens} {complex sentence}. Output: {simple sentence}," The control token and complex sentences Integrate into the Prompt template to form the Llama3.1 model input ,by As a label ,Right now: ； ； The resulting model inputs and labels are as follows: Example Input:"Below is an instruction that describes a task, paired withan input that provides further context. Write a response that appropriatelycompletes the request.### Instruction:Simplify the sentence given below byreferring to the control tokens.### Input:W_{ } C_{ } L_{ } WR_{ }DTD_{ } { }### Response:" Example Label: ; The Step 3 is specifically as follows: Using the trainer class inherited from the transformers library of HuggingFace, the calculation method of the training loss of the trainer is rewritten to obtain a dedicated trainer. During the model training process, the dedicated trainer first determines the training loss according to the new calculation method, and then uses the gradient information of the training loss to perform the back propagation algorithm to optimize and update the parameters of the model. The loss calculation method of the trainer is to calculate the standard loss of the simplified text generated by the model and the custom control token loss, that is: ； in, is the standard loss for the model to generate simplified text, which is the label And the simple sentence generated by the model The cross entropy loss between is: ； is the loss of the introduced control token, and the generated label is calculated Control Token And the simple sentence generated by the model Control Token The mean absolute error between them is: ； in, is the proportionality coefficient; The Step 4 is specifically as follows: Based on the unsloth framework, load the Llama3.1 model and the corresponding LoRA adapter, apply LoRA to the model, use the dedicated trainer, load the dedicated data set, and input the Llama3.1 model with the Prompt template. The trainer can follow the configured information and back-propagate to update the parameters of the low-rank matrix introduced by the LoRA method after each calculation of the training loss according to the loss function. After the training is completed, save the adjusted model weights to complete the model adjustment.

2. The controllable text simplification method for English subject graded reading according to claim 1 is characterized in that: The Step 5 is specifically as follows: For a new complex sentence , the model generates simple sentences The control token is set to , put the new complex sentence and the control token into the Prompt template to obtain the input of the model ,Right now: ； Will Input the adjusted Llama3.1 model, use the model generation method to get the model output token sequence, and then use the Llama3.1 model tokenizer to decode this token sequence to get the output simple sentence : =Fine-tuned Llama3.1 ; Among them, Fine-tuned Llama3.1 Indicates that Input the adjusted Llama3.1 model to get the output .

Images