A method for generating Chinese small samples based on temperature control fine-tuning model

Abstract

This invention relates to the field of news text augmentation and generation technology, and particularly to a method for generating Chinese few-shot text using a temperature-controlled fine-tuning model. The method includes: constructing high-quality news data text; obtaining a dataset of similar sentences; fine-tuning a pre-trained Transformer Block to improve its performance on similar sentence tasks, employing a fine-tuning method for the generative model to enhance the news expression ability of the Transformer Block model; constructing a Transformer Block generator; and using temperature control to generate news text. Based on the Transformer Block, a large amount of news text is generated. By continuously increasing the temperature to enhance the divergence of the news text, the richness of the generated news text is increased while maintaining its quality, ensuring that the generated news text meets the expected results.

Description

Technical Field

[0001] This invention relates to the field of news text augmentation and generation technology, and in particular to a method for generating small Chinese samples based on a temperature-controlled fine-tuning model. Background Technology

[0002] Text augmentation is an upstream task that generates variants of the original text using various methods, thereby increasing the diversity and quantity of training data. Text augmentation can effectively expand the size of the training dataset, allowing the model to see more different forms of input, which helps reduce overfitting. In situations where a large amount of labeled data is lacking, text augmentation can generate more training data, thus reducing reliance on manual annotation. Furthermore, the augmented text contains more grammatical variations, synonym substitutions, and word order adjustments, making the model more robust and better able to cope with changes and uncertainties in real-world environments.

[0003] Current text augmentation methods mainly include: First, EDA (Easy Data Augmentation), which increases the amount of text through random replacement, insertion, swapping, and deletion. Second, AEDA (AnEasier Data Augmentation), which enhances data by adding punctuation marks in the middle of sentences. Third, back-translation, which augments text by translating it into another language and then back. Fourth, masked language models, which use Transformer Block models like BERT and UniLM to partially mask the original sentence, allowing the model to predict the masked portion and thus obtain a new sentence, completing the pre-training task. Fifth, injecting noise into the text to generate new text. The last method is generating news sentences based on large models. When generating text using these methods, methods one, two, three, and five modify only a very small part of the original sample, resulting in highly similar news sentences, which may lead to no performance improvement in downstream applications. Methods four and six require a large amount of data during the model pre-training stage. In Chinese news generation scenarios, whether the generated news is complete in terms of main content and whether the semantics between texts are not missing are key factors in judging the quality of the generated samples. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention proposes a temperature-controlled fine-tuning model for generating small Chinese samples, thereby improving the quantity and quality of generated news texts.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for generating Chinese few samples based on a temperature-controlled fine-tuning model includes the following steps:

[0007] Step 1: Obtain the complete news dataset and perform preliminary cleaning on the obtained news dataset. At the same time, obtain the pre-trained weights for model initialization.

[0008] Step 2: Obtain a dataset of similar sentences to form a large news dataset;

[0009] Step 3: Pre-train the Transformer Block model using a similar sentence dataset, and then adjust the model parameters through backpropagation by calculating the loss.

[0010] Step 4: Input the news text of the desired augmentation domain into the pre-trained Transformer Block model, and use temperature control to generate the news text data;

[0011] Step 5: Filter the generated news text data to obtain the news text augmentation dataset.

[0012] Furthermore, the news dataset mentioned in step 1 is a public news text dataset containing several data entries; each news text entry contains a news title, a news body, and a news category.

[0013] The preliminary cleaning of the acquired news dataset involves performing a preliminary deduplication check on the entire news dataset to remove irrelevant information, including garbled text and advertisements.

[0014] The pre-trained weights used for model initialization are pre-trained parameters of BERT and UniLM models, or pre-trained publicly available model parameters.

[0015] Furthermore, the similar sentence dataset in step 2 contains several sets of similar text data; each set of data contains two texts and a similarity result between the texts, where 1 indicates similarity and 0 indicates dissimilarity.

[0016] Furthermore, step 3 specifically includes the following processes:

[0017] 3.1 Input the similar sentence dataset S into the Transformer Block model, perform similarity data discrimination, and obtain a discrimination result of length l. sim The loss is calculated using the SmoothL1Loss loss function based on the discrimination results. sim ;

[0018] result sim ={BERT(s) n.1 ,s n.2 ),n=1,2…l}

[0019] Wherein, BERT is the Transformer Block model using a BERT-like structure, n is the number of the selected text similarity pair, l is the length of the discrimination result, (s n.1 ,s n.2 () represents similar text pairs;

[0020] 3.2 Input the similar sentence dataset S into the Transformer Block model to generate similar text, and obtain a generated result of length l. out The loss is calculated using the Cross Entropy Loss function on the generated results. out ;

[0021] result out ={BERT(s) n.1 [0, len(s)-2]), n=1,2…l}

[0022] Where len(s) refers to the length of the original text;

[0023] 3.3. For the calculated loss sim and loss out The values ​​are summed to obtain the true loss, and then backpropagation is performed to adjust the model parameters.

[0024] Furthermore, the temperature control method is a way to adjust the randomness of generated text during language model training. Temperature is a parameter used to adjust the probability of the model predicting the distribution of the next word. The probability calculation of each word after temperature adjustment is as follows:

[0025]

[0026] Where p(w) refers to the original generation probability of each word, p(w_new) refers to the generation probability of each word after temperature adjustment, and T is the temperature control parameter.

[0027] Furthermore, the specific steps for filtering the generated news text data are as follows:

[0028] The embedded representations are calculated using news text data generated by the trained Transformer Block model. The similarity is then calculated and sorted based on the embedded representations to obtain the k most similar synonyms to the input text; k is a set value.

[0029] The advantages of this invention compared to the prior art are:

[0030] 1. Core mechanisms for enhancing the diversity of generative models

[0031] Temperature control is an important generative model modulation technique that directly affects the diversity and randomness of generated results by adjusting the shape of the output probability distribution. In practical applications, temperature parameters alter the sampling process by smoothing or enhancing the probability distribution. When the temperature parameter is low, the model tends to select outputs with higher probabilities, thus generating more focused text that conforms to conventional expressions; when the temperature parameter is high, the model considers lower-probability options more often.

[0032] 2. Powerful text generation capabilities to meet the needs of multiple scenarios

[0033] In text generation tasks, adjusting temperature parameters can generate text content in various styles, ranging from conventional to innovative expressions. Furthermore, in automatic summarization and machine translation tasks, it can help generate more diverse outputs, providing more options for specific scenarios.

[0034] 3. Its role in data augmentation tasks

[0035] Temperature control, in data augmentation tasks, generates a large amount of differentiated text data, providing a rich data source for model training, increasing data diversity, and enhancing the model's generalization ability. By appropriately adjusting temperature parameters, more differentiated text can be generated during the generation phase, providing a rich data foundation and creative support for subsequent work. Attached Figure Description

[0036] Figure 1 This is a flowchart of the Chinese small sample generation method for a temperature-controlled fine-tuning model in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the Chinese small sample generation method based on temperature control in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] The temperature-controlled fine-tuning model Chinese small sample generation method of this embodiment, such as... Figure 1 As shown, the method includes the following steps:

[0040] Step 1: Obtain the complete news dataset and perform preliminary cleaning based on the obtained news dataset, while obtaining the pre-trained weights for model initialization;

[0041] The news dataset is a publicly available collection of news text data containing several data entries; each data entry contains a news title, a news body, and a news category.

[0042] The preliminary cleaning of the acquired news dataset involves performing a preliminary deduplication check on the entire news dataset, removing garbled characters, advertisements, and other irrelevant information.

[0043] In this implementation plan, the publicly available news text dataset mainly consists of news reports from multiple well-known Chinese news media outlets, covering various aspects such as politics, military affairs, and society. Meanwhile, the model used is a pre-trained weight set of the open-source UniLM or SimBERT model downloaded from the HuggingFace official website, which contains 12 layers, 768 hidden layers, and 12 attention heads.

[0044] Step 2: Obtain a dataset of similar sentences to form a large news dataset;

[0045] The similar sentence dataset contains similar text data in rows; each row contains two texts and a similarity result between the texts, where 1 indicates similarity and 0 indicates dissimilarity.

[0046] The similar texts are derived from 22 million similar statements on Baidu Knows.

[0047] In this embodiment, considering the significant differences between texts, we first construct ({s n.1 s n.2 For text pairs, the input text is padded to a specified length, which is 512 tokens. At the same time, an AttentionMask is designed. The AttentionMask sets the text to 1 and the padded text to 0, so that the model knows where the text is and where the padded tokens are.

[0048] Step 3: Fine-tune the pre-trained TransformerBlock model using a similar sentence dataset to improve its performance in understanding similar sentences. Simultaneously, fine-tune it using a large amount of news text, employing generative model fine-tuning methods to enhance the TransformerBlock model's news expression capabilities.

[0049] In the following steps, 3.1 and 3.2 use the Transformer Block model for the data discrimination task and the text generation task, respectively. The difference is that the data discrimination task is a pre-training task, while the text generation task is a model fine-tuning task. The specific process includes:

[0050] Step 3.1: Input the similar sentence dataset S into the TransformerBlock model, perform similarity data discrimination, and obtain the result of length l. sim The loss is calculated using the SmoothL1Loss loss function on the generated results. sim ;

[0051] Furthermore, the aforementioned similar sentence dataset S refers to the dataset containing similar sentences paired with s. n.1 s n.2 ∈S, consider {s} when inputting. n.1 s n.2};

[0052] The result sim This refers to:

[0053] result sim ={BERT(s) n.1 s n.2 ), n = 1, 2, ..., l}

[0054] Where BERT refers to the TransformerBlock model with a BERT-like structure, n represents the number of the selected text similarity pair, and result... sim It refers to the calculated result.

[0055] SmoothL1Loss refers to a commonly used loss function, typically used in regression tasks. Its advantage over the Mean Squared Error (MSE) loss function lies in its smaller penalty for outliers (such as excessively large or small outliers), thus making the model more robust. Its expression is as follows:

[0056]

[0057] Where x refers to the predicted text, f(x) refers to the result predicted by the model, that is, the label that the model thinks it is, and y is the actual label.

[0058] Step 3.2: Input the similar sentence dataset S into the TransformerBlock model to generate similar text, and obtain a result of length l. out The loss is calculated using Cross Entropy Loss on the generated results. out ;

[0059] The results obtained from the pre-trained model refer to:

[0060] result out ={BERT(s) n.1[0, len(s)-2]), n=1, 2…l}

[0061] Where len(s) refers to the length of the original text.

[0062] The Cross Entropy Loss refers to a commonly used loss function, typically used to determine how close the actual output is to the expected output. Its expression is as follows:

[0063] Cross Entry Loss(p, q)=-∑ x p(x)log(q(x))+(1-p(x))log(1-q(x))

[0064] Where the distribution p(x) is the expected output and the probability distribution q(x) is the actual output.

[0065] Step 3.3: For the calculated loss sim and loss out The results are summed to obtain the true loss, which is then backpropagated. Minimizing the loss can improve the model's expressive power.

[0066] In this implementation scheme, the loss calculation locations involved in steps 3.1 and 3.2 are as follows: Figure 2 As shown.

[0067] In this implementation, a unified approach to various natural language processing tasks is achieved by sharing the same Transformer Block architecture and combining it with different task objectives. During training, the model simulates different task scenarios using different masking strategies. Simultaneously, by pre-training the same model using different masking strategies, the model can share parameters and knowledge across different tasks. Ultimately, by employing a unified Transformer Block architecture and multiple masking strategies, unified processing of various natural language processing tasks is achieved.

[0068] In this implementation, training the model using a joint loss function allows for the sharing of knowledge and representations across multiple tasks. This means that useful features learned in one task can help other related tasks. Multi-task learning helps prevent overfitting. By learning from multiple tasks simultaneously, the model avoids overfitting to data from a single task, as it must weigh trade-offs across multiple tasks. This sharing mechanism acts as a regularization mechanism, reducing the model's dependence on specific patterns from a single task. Multi-task learning often improves the model's generalization ability, enabling it to perform better on unseen data. This is because the model learns more general features and representations through multi-task learning.

[0069] In this embodiment, the process of fine-tuning the model is as follows: input the training samples into the model, train it, and stop training when the loss value decreases slightly while the accuracy and recall are high.

[0070] Step 4: Input the fine-tuned parameters into the Transformer Block model;

[0071] In this embodiment, the model trained in step 3 is deployed to downstream tasks;

[0072] Step 5: Input the news text of the desired augmentation area into the fine-tuned model, and use temperature control to generate the news text data;

[0073] The temperature control method mentioned in step 5 involves adjusting the randomness of generated text during language model training. Here, "temperature" refers to a parameter that controls the determinism and randomness of the model's output during generation. By adjusting the temperature parameter, text can be generated from more deterministic to more random text.

[0074] When generating text, the language model predicts the probability distribution of the next possible word. The temperature parameter adjusts this probability distribution, allowing the generated text to be either more random or more deterministic.

[0075] The probability calculations for each word after temperature adjustment are as follows:

[0076]

[0077] Where p(w) refers to the original generation probability of each word, p(w_new) refers to the generation probability of each word after temperature adjustment, and T is the temperature control parameter.

[0078] In this embodiment, text is input into a pre-trained model, and different generation results can be obtained. In this embodiment, it is considered that the text generated at T=0.9 has a higher similarity, while the text generated at T=1.38 is more discrete, but the semantics and similarity between the characters still remain at a high level.

[0079] Step 6: Filter the generated news text data to obtain the news text augmentation dataset;

[0080] The filtering of the generated news text data in step 6 refers to encoding the text generated by the model and calculating their embedding representations, calculating similarity based on the embedding representations and sorting them, and returning the k generated sentences that are most similar to the input text.

[0081] When calculating similarity, a pre-trained Transformer Block model is used to extract the embedding vector for each sentence. The embedding vectors are then L2 regularized (normalized). Finally, the cosine similarity between each generated sentence and the original text is calculated.

[0082] Emb = BERT(Input, T)

[0083]

[0084] Here, "Emb" refers to the sentence vector encoded using the model. L2 is the regularization term, and L2 is the normalized vector encoding. x and y refer to the original text encoding and the generated text encoding after encoding and normalization.

[0085] In this implementation, different values ​​of T are used to construct augmented sets of news samples. After inputting the text, a large number of encodings of similar sentences are generated. L2 regularization is used to add a penalty term to the model's loss function to make the learned model parameters more robust and to combat overfitting. Finally, an augmented set of the target sentence, Extend, is generated. input ;

[0086] Extend input ={s1, s2, ...}

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for generating Chinese small sample sizes using a temperature-controlled fine-tuning model, characterized in that, Includes the following steps: Step 1: Obtain the complete news dataset and perform preliminary cleaning on the obtained news dataset. At the same time, obtain the pre-trained weights for model initialization. Step 2: Obtain a dataset of similar sentences to form a large news dataset; Step 3: Pre-train the Transformer Block model using a similar sentence dataset, and then adjust the model parameters through backpropagation by calculating the loss. Step 4: Input the news text of the desired augmentation domain into the pre-trained Transformer Block model, and use temperature control to generate the news text data; Step 5: Filter the generated news text data to obtain the news text augmentation dataset.

2. The method for generating Chinese small sample sizes based on temperature control for fine-tuning models according to claim 1, characterized in that, The news dataset mentioned in step 1 is a public news text dataset containing several data entries; each news text entry contains a news title, a news body, and a news category. The preliminary cleaning of the acquired news dataset involves performing a preliminary deduplication check on the entire news dataset to remove irrelevant information, including garbled text and advertisements. The pre-trained weights used for model initialization are pre-trained public model parameters, including BERT-like and UniLM model pre-trained parameters.

3. The method for generating Chinese small samples based on temperature control fine-tuning model according to claim 1, characterized in that, The similar sentence dataset in step 2 contains several similar text data; each data contains two texts and a similarity result between the texts, where 1 indicates similarity and 0 indicates dissimilarity.

4. The method for generating Chinese small samples based on a temperature-controlled fine-tuning model according to claim 3, characterized in that, Step 3 includes the following specific steps: 3.1 Input the similar sentence dataset S into the Transformer Block model, perform similarity data discrimination, and obtain a discrimination result of length l. sim The loss is calculated using the SmoothL1Loss loss function based on the discrimination results. sim ; result sim ={BERT(s n.1 ,s n.2 ),n=1,2…l} Wherein, BERT is the Transformer Block model using a BERT-like structure, n is the number of the selected text similarity pair, l is the length of the discrimination result, (s n.1 ,s n.2 () represents similar text pairs; 3.2 Input the similar sentence dataset S into the Transformer Block model to generate similar text, and obtain a generated result of length l. out The loss is calculated using the Cross Entropy Loss function on the generated results. out ; result out ={BERT(s n.1 [0,len(s)-2]),n=1,2…l} Where len(s) refers to the length of the original text; 3.

3. For the calculated loss sim and loss out The values ​​are summed to obtain the true loss, and then backpropagation is performed to adjust the model parameters.

5. The method for generating Chinese small samples based on temperature control for fine-tuning models according to claim 3, characterized in that, The temperature control method described is a technique for adjusting the randomness of generated text during language model training. Temperature is a parameter used to adjust the probability of the model predicting the distribution of the next word. The probability calculation of each word after temperature adjustment is as follows: Where p(w) refers to the original generation probability of each word, p(w_new) refers to the generation probability of each word after temperature adjustment, and T is the temperature.

6. The method for generating Chinese small sample sizes based on temperature control for fine-tuning models according to claim 4, characterized in that, The specific steps for filtering the generated news text data are as follows: The embedded representations are calculated using news text data generated by the trained Transformer Block model. The similarity is then calculated and sorted based on the embedded representations to obtain the k most similar synonyms to the input text; k is a set value.

Images