Style text generation model training method and style text generation method
By obtaining social account information and historical posts, building a training sample set and training a large language model, the problems of generation stability and personalization are solved, and the style-consistent text generation is achieved.
Patent Information
- Application Number
- CN202510492862.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-18
- Publication Date
- 2025-08-15
AI Technical Summary
The prior art is difficult to generate text that is consistent with the style of a particular character, especially in the absence of parallel samples, where the text generation effect is unstable and uncontrollable.
By obtaining the basic information and historical posts of the target person's social account, extracting the title, summary and core content, building a training sample set, and using multi-category cross-entropy loss function to train pre-trained large language models to generate a style text generation model.
It improves the accuracy and personalization of text generation, enhances the generalization ability and robustness of the model, and makes the generated text consistent and stable with the style of the target character.
Smart Images
Figure CN120493903A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a style text generation model training method and a style text generation method, which are applicable to the field of information processing technology. Background Art
[0002] Styled text generation involves modifying specific attributes of text without changing its content. Styled text generation primarily focuses on generating text with attributes like sentiment and tense. This type of styled text generation typically involves training a sequence-to-sequence model on a set of paired training examples. By inputting text of one style into the model, the model generates text of another style.
[0003] Generating text based on person attributes is challenging due to the paired data of parallel samples and the complex and diverse nature of person attributes. Some techniques collect post data from specific people and train Markov chains or recurrent neural network models to generate text that closely resembles their posts. However, these models use input from previous words to predict subsequent text, resulting in a complete text. This makes the generated content uncontrollable and unpredictable, and the quality of the text is also unstable. Summary of the Invention
[0004] The technical problem to be solved by the present invention is: to address the above-mentioned problems, a style text generation model training method and a style text generation method are provided.
[0005] The technical solution adopted by the present invention is: a style text generation model training method, characterized by comprising:
[0006] S1. Obtain the target person's social account basic information and historical posts, and use multiple extraction methods to extract titles, summaries, and core content from historical posts;
[0007] S2. Based on the title, summary or core content, combined with the corresponding historical posts and basic information of social accounts, a training sample set is constructed;
[0008] S3. Based on the training sample set, train the pre-trained large language model to obtain a trained style text generation model.
[0009] The basic information of the social account includes one or more of the following: account name, number of fans, number of following, account authentication type, age, and account description.
[0010] The basic information of the social accounts used to construct the training sample set is the basic information of the social accounts obtained in step S1 or an arrangement and combination of various fields based on the basic information of the social accounts obtained in step S1.
[0011] The core content is automatically extracted by using a named entity recognition model on historical posts to obtain the corresponding core content, including the 5W1H entities.
[0012] The loss function in step S3 uses a multi-category cross entropy loss function. The number of categories is the dictionary size of the large language model. The specific expression is as follows:
[0013]
[0014] Where N is the number of characters in the predicted post, M is the dictionary size of the pre-trained language model, and y ic Is a sign function (value 0 or 1), if the true value of sample i is equal to c, then y ic =1, otherwise y ic =0, p ic It is the probability value of the predicted sample i taking the value c.
[0015] A style text generation model training device, characterized by comprising:
[0016] The data acquisition module is used to obtain the target person's social account basic information and historical posts, and use multiple extraction methods to extract titles, summaries and core content from historical posts;
[0017] The sample construction module is used to construct a training sample set based on the title, summary or core content, combined with the corresponding historical posts and basic information of social accounts;
[0018] The model training module is used to train the pre-trained large language model based on the training sample set to obtain a trained style text generation model.
[0019] A storage medium stores a computer program that can be executed by a processor, characterized in that the steps of the training method are implemented when the computer program is executed.
[0020] A style text generation model training device comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and is characterized in that the steps of the training method are implemented when the computer program is executed.
[0021] A style text generation model, characterized by being trained using the style text generation model training method.
[0022] A method for generating style text, characterized by:
[0023] Obtain the target person's basic social account information and the basic text of the text to be generated;
[0024] The basic text and basic information of the social account are input into the style text generation model trained by the style text generation model training method, and the generated text is output.
[0025] The beneficial effects of the present invention are as follows: the present invention collects the basic information and historical post data of the target person's social account, and obtains the corresponding title, summary, core content, etc. based on the historical post data. The basic information of the target person is encoded by using a transformer encoder to obtain a style-encoded latent vector, which is used as the starting character vector of the decoder. The title, summary, and core content are combined as the content input of the decoder, and finally decoded to obtain an output that is consistent with the style of the target person's social account and the content is consistent with the input.
[0026] To improve the generalization of the model and increase the amount of data, this paper uses different extraction models and parameter configurations to extract titles, abstracts, and core content with similar content as a data augmentation method. Furthermore, the paper constructs multiple instruction templates and paired parallel sample data, which also increases the amount of data and improves the generalization of the model.
[0027] By combining basic information and summaries from social media accounts, the model can obtain more comprehensive contextual information, which helps generate more accurate and personalized text. This comprehensive information can provide more clues to help the model understand the target person's preferences, style, and language habits.
[0028] In this paper, the basic information of social accounts provides the model with background information about the target person, while the summary condenses the key information. This combination of inputs helps generate higher-quality text content. The model can better capture the target person's language characteristics and topic trends, thereby improving the relevance and accuracy of the generated text.
[0029] This invention uses basic information and summaries as input, helping the model maintain better adaptability and generalization across diverse topics and scenarios. Rather than relying solely on direct imitation of historical posts, the model can incorporate more information to generate text, making the generated text more flexible and diverse.
[0030] During training, the basic information of social media accounts provides a stable reference, while the summary provides dynamically changing information. This combination of static and dynamic information helps improve the model's robustness in the face of new situations, making it more stable and reliable when generating text. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 This is a training flow chart in the embodiment.
[0032] Figure 2Generate an inference flowchart for the style text in the embodiment. DETAILED DESCRIPTION
[0033] Example 1: This example is a method for training a style text generation model, which specifically includes the following steps:
[0034] S1. Obtain the target person’s social account basic information and historical posts, and use multiple extraction methods to extract titles, summaries, and core content from historical posts.
[0035] In this example, the basic information of the social account includes fields such as account name, number of fans, number of following, account authentication type, age, and account description. If there is no corresponding field, it will be empty.
[0036] In this embodiment, a title extraction model is used for historical posts to automatically extract the corresponding title content; a summary extraction model is used for historical posts to automatically extract the corresponding summary content; and a named entity recognition model is used for historical posts to automatically extract the corresponding core content, including the 5W1H entities (Who, Where, When, What, Why, How).
[0037] In order to increase the training data, different models and different configurations are used to extract titles, summaries, and core contents respectively, so that a historical post corresponds to multiple titles, summaries, and core contents with similar content.
[0038] To increase training data, this example performs different permutations and combinations on the non-empty fields in the basic information of social accounts to construct basic information with multiple fields shuffled.
[0039] S2. Based on the title, summary or core content, combined with the corresponding historical posts and basic information of social accounts, a training sample set is constructed.
[0040] In this embodiment, the basic information of the social account is combined with the title, summary, and core content to construct input instructions. Multiple instruction templates are set as follows, which serve as parallel sample data of input-output pairs of the large language model:
[0041] Instruction template 1:
[0042] Style input (corresponding to basic social account information): You are XXX (social platform: Weibo, etc.) user XXX, with XXX followers and XXX followings. You are XXX (authentication type: certified / uncertified) user, XX years old, and you are a XXXXXX (user description: such as a senior media professional...)
[0043] Content input: Please publish a post on XXX (social platform: Weibo, etc.) with XXXXX (title / summary / core content);
[0044] Concatenate the two input formats together as input text:
[0045] You are XXX (social platform: Weibo, etc.) user XXX, with XXX followers and XXX followings. You are XXX (authentication type: certified / uncertified), age XX, and a XXXXXX (user description: such as a senior media professional...). Please post a XXX (social platform: Weibo, etc.) post in the style of XXX (user name) and with XXXXX (title / summary / core content).
[0046] Output: original post (corresponding historical post)
[0047] Instruction template 2:
[0048] Style input: You are XXX (social platform: Weibo, etc.) user XXX, the number of fans is XXX, the number of followers is XXX, you are XXX (authentication type: certified / non-certified) user, age XX years old, you are a XXXXXX (user description: such as senior media person...), please use the style of XXX (user name)
[0049] Content input: Publish a post on XXX (social platform: Weibo, etc.) based on XXXXX (title / abstract / core content);
[0050] Concatenate the two input formats together as input text:
[0051] You are user XXX on XXX (social platform: Weibo, etc.), with XXX followers and XXX followings. You are user XXX (authentication type: authenticated / unauthenticated), XX years old, and a XXXXXX (user description: e.g., a senior media professional). Please post a post on XXX (social platform: Weibo, etc.) in the style of XXX (user name) using XXXXX (title / summary / core content).
[0052] Output: original post (corresponding historical post)
[0053] S3. Based on the training sample set, train the pre-trained large language model to obtain a trained style text generation model.
[0054] The input text in the training sample set is used as the input of the model. The text is segmented and encoded into a numerical form that the model can understand using AutoTokenizer. The encoding is then input into the Transformer model for word embedding and multi-head attention mechanism to finally obtain a latent vector.
[0055] Pre-training large language models usually uses a decoder to predict the next character based on the previous content through autoregression, that is, at the first position according to the default starting character <s>, and get the hidden vector h, predict the character s1 at the first position according to the hidden vector, and the character s2 at the second position according to <s>s1 predicts the character s2 at the second position, and so on. The above methods are all performed through the transformer decoder.
[0056] In this embodiment, the transformer encoder is used to encode the style input to obtain the final latent vector h of the style input, and then the latent vector is used as the input of the first position of the pre-trained language model content decoder, and the default starting character is no longer used. <s>The latent vector of the text is gradually predicted by self-encoding, and the output characters are predicted based on the previous content. When calculating the loss function, only the loss function of the predicted output characters is calculated, and the input characters use the real value.
[0057] In this example, the loss function uses a multi-category cross entropy loss function. The number of categories is the dictionary size of the large language model. The specific expression is as follows:
[0058]
[0059] Where N is the number of characters in the predicted post, M is the dictionary size of the pre-trained language model, and y ic Is a sign function (value 0 or 1), if the true value of sample i is equal to c, then y ic =1, otherwise y ic =0, p ic It is the probability value of the predicted sample i taking the value c.
[0060] Example 2: This example is a style text generation model training device, including: a data acquisition module, a sample construction module and a model training module.
[0061] In this example, the data acquisition module is used to obtain the basic information and historical posts of the target person's social account, and uses multiple extraction methods to extract titles, summaries and core content from historical posts.
[0062] The sample construction module is used to construct a training sample set based on the title, summary or core content, combined with the corresponding historical posts and basic information of social accounts.
[0063] In this embodiment, the model training module is used to train a pre-trained large language model based on a training sample set to obtain a trained style text generation model.
[0064] Example 3: This example is a storage medium on which a computer program that can be executed by a processor is stored. When the computer program is executed, the steps of the training method in Example 1 are implemented.
[0065] Example 4: This example is a style text generation model training device, which has a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed, the steps of the training method in Example 1 are implemented.
[0066] Example 5: This example is a style text generation model, which is trained using the style text generation model training method in Example 1.
[0067] Example 6: This example is a method for generating style text, which specifically includes the following steps:
[0068] Obtain the target person's basic social account information and the basic text of the text to be generated;
[0069] The basic text (which may be a title, abstract or core content) and basic information of the social account are input into the style text generation model trained by the style text generation model training method in Example 1, and the generated text is output.< / s> < / s> < / s>
Claims
1. A style text generation model training method, characterized by: include: S1. Obtain the target person's social account basic information and historical posts, and use multiple extraction methods to extract titles, summaries, and core content from historical posts; S2. Based on the title, summary or core content, combined with the corresponding historical posts and basic information of social accounts, a training sample set is constructed; S3. Based on the training sample set, train the pre-trained large language model to obtain a trained style text generation model.
2. The method for training a style text generation model according to claim 1, wherein: The basic information of the social account includes one or more of the following: account name, number of fans, number of following, account authentication type, age, and account description.
3. The method for training a style text generation model according to claim 2, wherein: The basic information of the social account used to construct the training sample set is the basic information of the social account obtained in step S1 or an arrangement and combination of various fields based on the basic information of the social account obtained in step S1.
4. The method for training a style text generation model according to claim 1, wherein: The core content is automatically extracted by using a named entity recognition model on historical posts to identify the corresponding core content, including the 5W1H entities.
5. The method for training a style text generation model according to claim 1, wherein: The loss function in step S3 uses a multi-category cross entropy loss function. The number of categories is the dictionary size of the large language model. The specific expression is as follows: Where N is the number of characters in the predicted post, M is the dictionary size of the pre-trained language model, and y ic Is a sign function (value 0 or 1), if the true value of sample i is equal to c, then y ic =1, otherwise y ic =0, p ic It is the probability value of the predicted sample i taking the value c.
6. A style text generation model training device, characterized in that: include: The data acquisition module is used to obtain the target person's social account basic information and historical posts, and use multiple extraction methods to extract titles, summaries and core content from historical posts; The sample construction module is used to construct a training sample set based on the title, summary or core content, combined with the corresponding historical posts and basic information of social accounts; The model training module is used to train the pre-trained large language model based on the training sample set to obtain a trained style text generation model.
7. A storage medium having stored thereon a computer program executable by a processor, characterized in that: When the computer program is executed, the steps of the training method according to any one of claims 1 to 5 are implemented.
8. A device for training a style text generation model, comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, and wherein: When the computer program is executed, the steps of the training method according to any one of claims 1 to 5 are implemented.
9. A style text generation model, characterized by: The style text generation model is obtained by training using the style text generation model training method described in any one of claims 1 to 5.
10. A method for generating style text, characterized by: Obtain the target person's basic social account information and the basic text of the text to be generated; The basic text and basic information of the social account are input into the style text generation model trained by the style text generation model training method according to any one of claims 1 to 5, and the generated text is output.