Fine-tuning model Chinese small sample generation method based on temperature control

Abstract

The invention relates to the technical field of news text augmentation and generation, in particular to a fine-tuning model Chinese small sample generation method based on temperature control. The method comprises the following steps: constructing a high-quality news data text; obtaining a similar sentence data set; according to the method, the pre-trained Transform Block is finely adjusted, so that the Transform Block is more excellent in similar sentence tasks, and the news expression ability of a Transform Block model is improved by adopting a fine adjustment method for generating a model; the method comprises the following steps of: constructing a Transform Block generator; and news text generation is realized by using a temperature control mode. A large number of news texts are generated based on Transform Block, the divergence of the news texts is improved by continuously increasing the temperature, the richness is increased while the quality of the generated news texts is not reduced, and it is ensured that the generated news texts meet the expected effect.

Description

Technical Field

[0001] The present invention relates to the technical field of news text augmentation and generation, and in particular to a method for generating Chinese small samples based on a fine-tuning model with temperature control. Background Art

[0002] Text augmentation is an upstream task that generates variants of the original text through various methods to increase the diversity and quantity of training data. Text augmentation can effectively expand the size of the training dataset, allowing the model to see more different forms of input, which helps reduce overfitting. In the absence of a large amount of annotated data, text augmentation can generate more training data, thereby reducing dependence on manual annotation. At the same time, the augmented text contains more grammatical variants, synonym replacements, and word order adjustments, making the model more robust and better able to cope with changes and uncertainties in the real environment.

[0003] There are several main methods for text augmentation at present: the first is EDA (Easy Data Augmentation), which increases the amount of text by random replacement, random insertion, random exchange, random deletion and other methods. The second is AEDA (AnEasier Data Augmentation), which enhances data by adding punctuation marks in the middle of sentences. The third is back translation, which achieves the purpose of augmenting text by translating text into another language and then translating it back. The fourth is the masked language model, that is, using Transformer Block model structures such as BERT and UniLM to partially mask the original sentence, let the model predict the masked part, and thus obtain a new sentence to complete the pre-training task. The fifth is to inject noise into the text to generate new text. The last one is to generate news sentences based on a large model. When generating text using the above methods, since methods one, two, three, and five all modify a very small part of the original sample, the generated news sentences are extremely similar, which may cause the augmented text to have no performance improvement in downstream task applications. For methods four and six, a large amount of data is required in the pre-training stage of the model. In the Chinese news generation scenario, whether the generated news is complete and whether there is no semantic gap between texts are the key to judging the quality of the generated samples. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a Chinese small sample generation method based on a fine-tuning model of temperature control to improve the generation quantity and quality of news texts.

[0005] The technical solution adopted by the present invention is:

[0006] A method for generating Chinese small samples of a fine-tuning model based on temperature control, comprising the following steps:

[0007] Step 1: Obtain the entire news dataset, perform preliminary cleaning on the obtained news dataset, and obtain the pre-trained weights for model initialization;

[0008] Step 2: Obtain similar sentence datasets to form a large number of news datasets;

[0009] Step 3: Pre-train the Transformer Block model using a similar sentence dataset and adjust the model parameters by back-propagating the calculated loss.

[0010] Step 4: Input the news text of the required augmented domain into the pre-trained Transformer Block model, and use temperature control to generate news text data;

[0011] Step 5: Filter the generated news text data to obtain the news text augmented data set.

[0012] Furthermore, the complete set of news data sets in step 1 is a public news text data set containing a number of pieces of data; each piece of news text data contains a news title, a news body and a news category;

[0013] The preliminary cleaning of the entire set of acquired news data is to perform a preliminary duplicate check on the entire set of news data and remove irrelevant information, including garbled codes and advertisements;

[0014] The pre-trained weights used for model initialization are pre-trained parameters of BERT-like and UniLM-like models, or pre-trained public model parameters.

[0015] Furthermore, the similar sentence data set in step 2 includes similar text data of several pieces of data; each piece of data includes two texts and a similarity result between the texts, 1 for similarity and 0 for dissimilarity.

[0016] Furthermore, the specific process of step 3 includes:

[0017] 3.1. Input similar sentence dataset S to Transformer Block model, perform similar data discrimination, and obtain discrimination result of length l sim ; For the discrimination result, the SmoothL1Loss loss function is used to calculate the loss sim ;

[0018] Result sim = {BERT(s n.1 ,s n.2 ),n=1,2…l}

[0019] Among them, BERT is a Transformer Block model using a BERT-like structure, n is the number of the selected text similarity pair, l is the length of the discrimination result, (s n.1 ,s n.2 ) are similar text pairs;

[0020] 3.2. Input similar sentence dataset S to Transformer Block model, generate similar text, and obtain the generated result with length l out , the Cross Entropy Loss function is used to calculate the loss for the generated result out ;

[0021] Result out = {BERT(s n.1 [0,len(s)-2]),n=1,2…l}

[0022] Where len(s) refers to the length of the original text;

[0023] 3.3. For the calculated loss sim and loss out Add them together to get the actual loss, and perform back propagation to adjust the model parameters.

[0024] Furthermore, the temperature control method is a method for adjusting the randomness of generated text when training a language model. The temperature is a parameter, and the temperature parameter is used to adjust the probability of the model predicting the distribution of the next word. The probability of each word appearing in this model after temperature adjustment is calculated as follows:

[0025]

[0026] Where p(w) refers to the original generation probability of each word, p(w_new) refers to the generation probability of each word after temperature adjustment, and T is the temperature control parameter.

[0027] Furthermore, the generated news text data is screened as follows:

[0028] The news text data generated by the trained Transformer Block model is used to calculate the embedding representation. The similarity is calculated and sorted based on the embedding representation to obtain the k synonyms that are most similar to the input text; k is a set value.

[0029] The advantages of the present invention compared with the prior art are:

[0030] 1. Core Mechanisms for Enhancing Diversity of Generative Models

[0031] Temperature control is an important generative model control technique that directly affects the diversity and randomness of the generated results by adjusting the shape of the output probability distribution. In practical applications, the temperature parameter changes the sampling process by smoothing or enhancing the probability distribution. When the temperature parameter is low, the model tends to choose outputs with higher probabilities, thereby generating more concentrated text that conforms to conventional expressions; when the temperature parameter is high, the model will consider more options with lower probabilities.

[0032] 2. Powerful text generation capabilities to meet the needs of multiple scenarios

[0033] In text generation tasks, by adjusting the temperature parameter, a variety of styles of text content can be generated, from conventional to innovative expressions. At the same time, in automatic summarization and machine translation tasks, it can help generate more diverse outputs and provide more options for specific scenarios.

[0034] 3. Role in data expansion tasks

[0035] Temperature control can provide a rich data source for model training by generating a large amount of differentiated text data in the data expansion task, improve the diversity of data, and enhance the generalization ability of the model. By properly adjusting the temperature parameters, more differentiated text can be generated in the generation stage, providing a rich data foundation and creative support for subsequent work. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for generating Chinese small samples in a fine-tuning model based on temperature control in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of a method for generating Chinese small samples in a fine-tuning model based on temperature control in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] The method for generating Chinese small samples in the fine-tuning model based on temperature control in this embodiment is as follows: Figure 1 As shown, the method comprises the following steps:

[0040] Step 1: Obtain the entire news dataset, perform preliminary cleaning based on the acquired news dataset, and obtain the pre-trained weights for model initialization;

[0041] The complete news data set is a public news text data set containing a number of pieces of data; each piece of data contains a news title, a news text and a news category;

[0042] The preliminary cleaning of the entire set of acquired news data is to perform a preliminary duplicate check on the entire set of news data, remove garbled characters, and remove irrelevant information such as advertisements;

[0043] In this implementation scheme, the public news text dataset mainly uses news reported by many well-known Chinese news media, covering politics, military, society and other aspects. At the same time, the model is the open source UniLM or SimBERT model pre-training weight downloaded from the HuggingFace official website, which contains 12 layers, 768 hidden layers, and 12 attention heads.

[0044] Step 2: Obtain similar sentence datasets to form a large number of news datasets;

[0045] The similar sentence data set includes similar text data of 10 pieces of data; each piece of data includes two texts and a similarity result between the two texts, 1 for similarity and 0 for dissimilarity;

[0046] The similar texts are derived from Baidu’s 22 million similar sentences.

[0047] In this embodiment, considering the large differences between texts, we first construct ({s n.1 ,s n.2 For the input text, pad it to the specified length, that is, 512 tokens. At the same time, AttentionMask should also be designed. AttentionMask means setting 1 to the part with text and 0 to the padded part, so that the model can understand where the text is and where the padded symbols are.

[0048] Step 3: Fine-tune the pre-trained TransformerBlock model through a similar sentence dataset to make it perform better in the task of understanding similar sentences. At the same time, fine-tune using a large amount of news text and adopt the fine-tuning method of the generative model to improve the news expression ability of the TransformerBlock model;

[0049] In the following steps, 3.1 and 3.2 use the Transformer Block model for data discrimination and text generation tasks respectively. The difference is that the data discrimination task is a pre-training task, while the text generation task is a model fine-tuning task. The specific process includes:

[0050] Step 3.1: Input similar sentence dataset S to TransformerBlock model, perform similar data discrimination, and obtain the result result with a discrimination length of l sim . For the generated results, the SmoothL1Loss loss function is used to calculate the loss sim ;

[0051] Furthermore, the similar sentence dataset S refers to the similar sentence pairs s in the similar sentence dataset. n.1 ,s n.2 ∈S, consider {s n.1 ,s n.2};

[0052] The result sim Refers to:

[0053] Result sim = {BERT(s n.1 ,s n.2 ), n=1, 2…1}

[0054] BERT is the TransformerBlock model with a BERT-like structure, n represents the number of the selected text similarity pairs, and result sim Refers to the calculated result.

[0055] The SmoothL1Loss refers to a commonly used loss function, which is usually used in regression tasks. Its advantage over the mean square error (MSE) loss function is that it imposes less penalty on outliers (such as outliers that are too large or too small), making the model more robust. Its expression is as follows:

[0056]

[0057] Where x refers to the predicted text, f(x) refers to the result predicted by the model, that is, the label considered by the model, and y is the actual label.

[0058] Step 3.2: Input similar sentence dataset S to TransformerBlock model to generate similar text and obtain the result with length l. out . For the generated results, use Cross Entropy Loss to calculate the loss out ;

[0059] The results obtained by the pre-training model refer to:

[0060] Result out = {BERT(s n.1[0, len(s)-2]), n=1, 2...l}

[0061] Where len(s) refers to the length of the original text.

[0062] The Cross Entropy Loss refers to a commonly used loss function, which is usually used to determine the degree of closeness between the actual output and the expected output. Its expression is as follows:

[0063] Cross Entry Loss(p, q)=-∑ x p(x)log(q(x))+(1-p(x))log(1-q(x))

[0064] Where distribution p(x) is the expected output and probability distribution q(x) is the actual output.

[0065] Step 3.3: For the calculated loss sim and loss out Add them together to get the real loss for back propagation. By minimizing the loss, the expressiveness of the model can be improved.

[0066] In this embodiment, the loss calculation locations involved in step 3.1 and step 3.2 are as follows: Figure 2 shown.

[0067] In this embodiment, the unification of multiple natural language processing tasks is mainly achieved by sharing the same Transformer Block architecture and combining different task objectives. During the training process, the model simulates different task scenarios through different masking strategies. At the same time, by using different masking strategies for pre-training in the same model, the model can share parameters and knowledge in different tasks. Finally, by adopting a unified Transformer Block architecture and multiple masking strategies, unified processing of multiple natural language processing tasks is achieved.

[0068] In this embodiment, the joint loss function is used to train the model, which allows knowledge and representations to be shared between multiple tasks. This means that useful features learned by one task can help other related tasks. Multi-task learning helps prevent overfitting. By learning multiple tasks at the same time, the model will not overfit the data of a single task because it must make trade-offs on multiple tasks. This sharing mechanism can act as a regularizer and reduce the model's reliance on the specific patterns of a single task. Multi-task learning can generally improve the generalization ability of the model, allowing it to perform better on unseen data. This is because the model learns more general features and representations through multi-task learning.

[0069] In this embodiment, the process of fine-tuning the model is: inputting training samples into the model, training, and stopping the training when the loss value decreases slightly and the accuracy and recall rate are high;

[0070] Step 4: Use the fine-tuned parameters to input into the Transformer Block model;

[0071] In this implementation, the model trained in step 3 is deployed to downstream tasks;

[0072] Step 5: Input the news text in the required augmented domain into the fine-tuned model and use temperature control to generate news text data;

[0073] The temperature control method referred to in step 5 is a method that involves adjusting the randomness of the generated text when training the language model. The "temperature" here refers to a parameter that controls the certainty and randomness of the model output during the generation process. By adjusting the temperature parameter, you can generate text from more certainty to more random text;

[0074] When generating text, the language model predicts the probability distribution of the next possible word. The temperature parameter adjusts this probability distribution so that the text generated by the model can be more random or more deterministic.

[0075] The probability of each word appearing in this model after temperature adjustment is calculated as follows:

[0076]

[0077] Where p(w) refers to the original generation probability of each word, p(w_new) refers to the generation probability of each word after temperature adjustment, and T is the temperature control parameter.

[0078] In this embodiment, the text is input into the pre-trained model, and after generation, different generation results can be obtained. In this embodiment, it is considered that the text generated at T = 0.9 has a greater similarity, and the text generated at T = 1.38 is more discrete, but the semantics and similarity between the texts are still maintained at a high level.

[0079] Step 6: Filter the generated news text data to obtain the news text augmented data set;

[0080] The screening of the generated news text data in step 6 refers to encoding the texts generated by the model and calculating their embedding representations, calculating the similarity and sorting according to the embedding representations, and returning the k generated sentences that are most similar to the input text;

[0081] When calculating the similarity, the trained Transformer Block model is used to extract the embedding vector of each sentence. The embedding vector is L2 regularized (normalized). The cosine similarity between each generated sentence and the original text is calculated.

[0082] Emb=BERT(Input,T)

[0083]

[0084] Here Emb refers to the sentence vector encoded using the model. is the regularization term, L2 is the normalized vector encoding, and x and y refer to the original text encoding and generated text encoding after encoding and normalization.

[0085] In this implementation, different Ts are used to construct an augmented set of news samples. After the Input text is input, a large number of encodings of similar sentences are generated. L2 regularization is used to add a penalty term to the model loss function to make the learned model parameter value smaller to deal with overfitting. Finally, an augmented set of target sentences can be generated. input ;

[0086] Extend input ={s1,s2....}

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for generating Chinese small samples in a fine-tuning model based on temperature control, characterized in that: The steps include: Step 1: Obtain the entire news dataset, perform preliminary cleaning on the obtained news dataset, and obtain the pre-trained weights for model initialization; Step 2: Obtain similar sentence datasets to form a large number of news datasets; Step 3: Pre-train the Transformer Block model using a similar sentence dataset and adjust the model parameters by back-propagating the calculated loss. Step 4: Input the news text of the required augmented domain into the pre-trained Transformer Block model, and use temperature control to generate news text data; Step 5: Filter the generated news text data to obtain the news text augmented data set.

2. The method for generating Chinese small samples in a fine-tuning model based on temperature control according to claim 1, characterized in that: The complete news data set in step 1 is a public news text data set containing a number of pieces of data; each piece of news text data contains a news title, a news body and a news category; The preliminary cleaning of the entire set of acquired news data is to perform a preliminary duplicate check on the entire set of news data and remove irrelevant information, including garbled codes and advertisements; The pre-trained weights used for model initialization are pre-trained parameters of BERT-like and UniLM-like models, or pre-trained public model parameters.

3. The method for generating Chinese small samples in a fine-tuning model based on temperature control according to claim 1, characterized in that: The similar sentence data set in step 2 contains similar text data of several pieces of data; each piece of data contains two texts and a similarity result between the texts, 1 for similarity and 0 for dissimilarity.

4. The method for generating Chinese small samples in a fine-tuning model based on temperature control according to claim 3 is characterized in that: Step 3 The specific process includes: 3.

1. Input similar sentence dataset S to Transformer Block model, perform similar data discrimination, and obtain discrimination result of length l sim ; For the discrimination result, the SmoothL1Loss loss function is used to calculate the loss sim ; result sim ={BERT(s n.1 ,s n.2 ),n=1,2…l} Among them, BERT is a Transformer Block model using a BERT-like structure, n is the number of the selected text similarity pair, l is the length of the discrimination result, (s n.1 ,s n.2 ) are similar text pairs; 3.

2. Input similar sentence dataset S to Transformer Block model, generate similar text, and obtain the generated result with length l out , the Cross Entropy Loss function is used to calculate the loss for the generated result out ; result out ={BERT(s n.1 [0,len(s)-2]),n=1,2…l} Where len(s) refers to the length of the original text; 3.

3. For the calculated loss sim and loss out Add them together to get the actual loss, and perform back propagation to adjust the model parameters.

5. According to claim 3, a method for generating Chinese small samples in a fine-tuning model based on temperature control is characterized in that: The temperature control method is a method for adjusting the randomness of generated text when training a language model. The temperature is a parameter, and the temperature parameter is used to adjust the probability of the model predicting the distribution of the next word. The probability of each word appearing in this model after temperature adjustment is calculated as follows: Where p(w) refers to the original generation probability of each word, p(w_new) refers to the generation probability of each word after temperature adjustment, and T is the temperature control parameter.

6. The method for generating Chinese small samples in a fine-tuning model based on temperature control according to claim 4, characterized in that: The generated news text data is filtered as follows: The news text data generated by the trained Transformer Block model is used to calculate the embedding representation. The similarity is calculated and sorted based on the embedding representation to obtain the k synonyms that are most similar to the input text; k is a set value.

Images