Method for generating poems and blessings based on large model

Abstract

The invention relates to a poem and blessing generation method based on a large model, and the method comprises the steps: 1) carrying out the preprocessing of poem and blessing data, and forming a large-scale training data set; 2) using a large pre-training language model as a generative model, and pre-training the generative model to obtain a pre-trained generative model; 3) constructing training data of instructions, poem sentences and blessings, and performing instruction fine tuning on the pre-trained generation model by using the training data to obtain a poem sentence and blessings generation model; and 4) directly inputting the poem and blessing theme and the instruction into the poem and blessing generation model to generate the poem and blessing related to the poem and blessing theme. According to the method, the processing links of theme extraction and keyword extension during input are removed, a large model is adopted, and instruction fine tuning adapting to the fields of poems and blessings is added, so that extraction and addition errors are avoided, the quality of the generated poems and blessings is ensured, and the poems and blessings meeting requirements can be generated according to instructions.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and relates to a method for generating verses and blessings, and specifically to a method for generating verses and blessings based on a large model. Background Art

[0002] In recent years, the use of artificial intelligence models to generate poems and greetings has become increasingly common. However, the models used previously were relatively small and had limited ability to generate language. The formats of poems and greetings have specific specifications and need to have a certain rhythm, and the text cannot be stiff or awkward.

[0003] At present, the existing methods of using small models to generate poems and blessings mainly include two parts: 1. Processing the theme requirements of the input poems and blessings; 2. Inputting the processed requirements into the small model generator for generation. That is, the theme requirements of the input poems and blessings need to be processed first, such as extracting keywords and adding expansion words, and then using the small model to generate. These methods all assume that: the extracted keywords are accurate, the added expansion words are also accurate, and the generation of the small model is coherent. However, such methods will encounter many problems in practical application, mainly reflected in that the existing keyword extraction is still wrong and ambiguous, and the added expansion words will bring more errors, so even with the strongest extraction and addition tools, it is possible to encounter keyword and expansion word errors; secondly, even if the keywords and expansion words are correct, the generation of the small model will have various problems such as incoherence, lack of rhythm, and lack of artistic conception. In addition, the small model can only support the trained poems and blessing forms, and cannot meet the requirements of user diversity.

[0004] Therefore, the current method of generating poems and greetings based on small models has the following shortcomings: 1. The performance in terms of rhythm, coherence and artistic conception is not good enough; 2. The generalization ability outside the domain is poor, and the effect becomes worse when applied to data other than training data. 3. It does not support the diverse requirements of users in actual applications. A model can usually only support one type of poem. The model that generates general poems cannot support the generation of hidden poems for users.

[0005] Therefore, in view of the defects in the above-mentioned prior art, it is necessary to develop a new method for generating poems and blessings. Summary of the invention

[0006] In order to overcome the defects of the prior art, the present invention proposes a method for generating poems and blessings based on a large model, which avoids errors in extraction and addition by removing the processing steps of extracting topics and expanding keywords during input, adopting a large model, and adding fine-tuning instructions adapted to the field of poems and blessings, thereby ensuring the quality of the generated poems and blessings, and being able to generate poems and blessings that meet the requirements according to the instructions.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A method for generating poems and blessings based on a large model, characterized by comprising the following steps:

[0009] 1) Data preprocessing: Preprocess a large amount of poetry and blessing data to form a large-scale training data set;

[0010] 2) Model pre-training: using a large pre-trained language model as a generative model, and training the generative model with the large-scale training data set to obtain a pre-trained generative model;

[0011] 3) Instruction fine-tuning: construct training data of instructions plus poems and blessings, and use this form of training data to fine-tune the pre-trained generation model to obtain a poem and blessing generation model;

[0012] 4) Verses and blessings generation: The verses and blessings themes and instructions are not pre-processed for theme extraction and keyword expansion, but are directly input into the verses and blessings generation model to generate verses and blessings related to the verses and blessings themes.

[0013] Preferably, the preprocessing in step 1) is to tokenize the verses and blessing data to segment the verses and blessing data into sequences of characters or words.

[0014] Preferably, in step 1), the large amount of poetry and blessing data is greater than ten thousand poetry and blessing data and the large-scale training data set includes more than ten thousand training data.

[0015] Preferably, in step 2), the large pre-trained language model used is a GLM model, that is, a generalized linear model.

[0016] Preferably, in step 2), the GLM model used is a GLM model with 10 billion parameters.

[0017] Preferably, in the step 2), during the training process, the generation model is optimized through incremental pre-training to improve its generation quality and generalization ability.

[0018] Preferably, the method for generating poems and blessings based on the large model further comprises:

[0019] 5) Screening and filtering of poems and blessings: The generated poems and blessings are screened and filtered through evaluation indicators to ensure that they meet the expected requirements.

[0020] Preferably, the evaluation indicators include coherence, rhythm and artistic conception.

[0021] Compared with the prior art, the method for generating poems and blessings based on a large model of the present invention has one or more of the following beneficial technical effects:

[0022] 1. The present invention uses a large pre-trained language model as a generation model, so that the generation capability of the poetry and blessing generation model is more universal, the generated content is more coherent and rich in rhythm and artistic conception, and has a strong out-of-domain generalization capability.

[0023] 2. The present invention eliminates the steps of preprocessing such as topic extraction and keyword expansion for the themes of poems and blessings, and directly inputs the themes of poems and blessings into the poem and blessing generation model, shortening the processing flow and reducing the possibility of introducing additional inaccurate information during the preprocessing process, thereby achieving better generation effects.

[0024] 3. The present invention uses the technology of instruction fine-tuning. In the training data, specific instructions are taken as input, and poems and blessings that meet the instructions are taken as output, so as to train the model so that the model has the ability to generate poems and blessings according to the specific requirements of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the method for generating poems and blessings based on a large model of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and embodiments, and the contents of the embodiments are not intended to limit the protection scope of the present invention.

[0027] In view of the shortcomings of the existing methods for generating verses and blessings, the present invention provides a method for generating verses and blessings based on a large model, which avoids errors in extraction and addition and ensures the quality of the generated verses and blessings by removing the input processing link and adopting a large model.

[0028] Figure 1 The flowchart of the method for generating poems and greetings based on a large model of the present invention is shown. Figure 1 As shown, the method for generating poems and blessings based on a large model of the present invention comprises the following steps:

[0029] 1. Data preprocessing.

[0030] A large amount of poetry and blessing data is preprocessed to form a large-scale training data set.

[0031] In the present invention, the pre-processing is to tokenize the verses and blessing data to segment the verses and blessing data into sequences of characters or words. For example, through tokenization, "I wish you a happy new year" is segmented into a sequence consisting of "I wish you a happy new year", "you", "New Year", and "Happy".

[0032] At the same time, in order to ensure a sufficient amount of training data to ensure the quality of training, in the present invention, preferably, the large amount of poetry and blessing data is more than 10,000 poetry and blessing data and the large-scale training data set includes more than 10,000 training data.

[0033] 2. Model pre-training.

[0034] Before model pre-training, the first step is to select a generative model. Unlike the prior art that uses a small model as a generative model, in the present invention, a large model is used as a generative model, that is, a large pre-trained language model is used as a generative model.

[0035] Preferably, the large pre-trained language model used is a GLM model, i.e., a generalized linear model. The GLM model is a language model based on the Transformer architecture and has more than 130 billion parameters.

[0036] Meanwhile, the GLM model has multiple versions, and different versions have different parameters. In the present invention, more preferably, the GLM model used is a GLM model with 10 billion parameters. The GLM model with this version of 10 billion parameters can achieve faster generation speed and lower hardware requirements.

[0037] After the generative model is selected, the generative model is pre-trained using the large-scale training data set to obtain a pre-trained generative model.

[0038] In the present invention, since a large pre-trained language model is used as a generation model, the generation capability of the poetry and blessing generation model is more universal, the generated content is more coherent and rich in rhythm and artistic conception, and has a strong out-of-domain generalization capability.

[0039] In addition, in the present invention, during the pre-training process, the generation model can be optimized through incremental pre-training, thereby improving its generation quality and generalization ability.

[0040] Incremental pretraining is a method of further training based on a pretrained model. In incremental pretraining, the previously pretrained model is called the base model, and the incremental pretraining stage is to continue updating and learning the model parameters based on the base model.

[0041] Generally, incremental pre-training is to further optimize the performance of the model in a specific domain or task. In incremental pre-training, data from a specific domain or data related to a task can be used and combined with the basic model for targeted training. This allows the model to adapt to the characteristics, semantics, or task requirements of a specific domain, thereby improving the performance of the model in that domain or task.

[0042] In the present invention, the process of incremental pre-training includes the following steps:

[0043] 1. Load the basic model: Load the pre-trained generative model into the training environment.

[0044] 2. Prepare incremental data: collect or prepare data related to the generation of poems and blessings, that is, incremental data. The incremental data can be text, image, audio and other data.

[0045] 3. Incremental training: Combine the generative model with incremental data and update and learn the parameters of the generative model through the training algorithm. This can include fine-tuning the parameters of the generative model or using some specific optimization strategies.

[0046] 4. Evaluation and adjustment: After the incremental pre-training is completed, the generated model is evaluated and verified, and adjusted and optimized according to the performance indicators.

[0047] In the present invention, through incremental pre-training, a large amount of incremental data of poems and blessings can be used on the generative model that has been trained using general corpus to continue training and optimizing the generative model, so that the generative model can perform better in the task of generating poems and blessings.

[0048] 3. Fine-tune instructions.

[0049] Although the pre-trained generative model overcomes some defects of the prior art and can realize the generation of poems and greetings, in practical applications, it does not support the diverse requirements of users. For example, the model that generates general poems for users cannot support the generation of acrostic poems. For this reason, in the present invention, the instruction fine-tuning technology is used to process the training data.

[0050] Specifically, training data of instructions plus poems and blessings is constructed, that is, the poems and blessings themes and instructions (that is, the specific requirements of the user) are taken as input, and the poems and blessings that meet the instructions (that is, the specific requirements of the user) are taken as output, so as to train the pre-trained generation model and obtain the poems and blessings generation model.

[0051] Since the training data of instructions plus poems and blessings is constructed, the poem and blessing generation model has the ability to generate poems and blessings according to user instructions (that is, the user's specific requirements). For example, acrostic poems can be generated.

[0052] 4. Generate poems and blessings.

[0053] Unlike the prior art which requires processing of the verses and blessing themes, such as extracting keywords, adding extension words, etc., in the present invention, there is no need to pre-process the verses and blessing themes. Instead, the verses and blessing themes and instructions are directly input into the verses and blessing generation model without any pre-processing of theme extraction and keyword expansion to generate verses and blessings related to the verses and blessing themes.

[0054] Since the processing link of the verses and blessing themes is removed and the verses and blessing themes are directly input into the verses and blessing generation model, the processing process is shortened, the possibility of introducing additional inaccurate information during the preprocessing process is reduced, and better generation effects can be obtained.

[0055] At the same time, the user inputs the theme of the poem and blessing, such as "birthday", "love", etc., and inputs specific requirements, such as "acrostic poem". The input poem and blessing theme and specific requirements will be directly input into the poem and blessing generation model, and no pre-processing is required. The poem and blessing generation model will generate coherent, rhythmic and artistic poems and blessings related to the poem and blessing theme and in accordance with the specific requirements of the user based on the input poem and blessing theme.

[0056] In the present invention, assuming that the theme of the poem and blessing input by the user is "Happy Year of the Rabbit", then "Happy Year of the Rabbit" is directly input into the poem and blessing generation model, and the poem and blessing generated by the poem and blessing generation model may be as follows: "I wish you good luck and peace, I wish you happiness and joy, I wish you health and beauty, I wish you good luck in the Year of the Rabbit, I wish you great fortune in the Year of the Rabbit."

[0057] Assuming that the theme of the poem and blessing input by the user is "Happy New Year to my brothers", and the specific input requirement is "hidden-head poem", then "Happy New Year to my brothers" and "hidden-head poem" are directly input into the poem and blessing generation model, and the hidden-head poem generated by the poem and blessing generation model may be as follows: "I wish you a thousand years of longevity, and I travel with you. I return my brother's horse, and we share a two-person carriage. In the place where we have fun in the New Year, we drink New Year wine in the bamboo forest to the west. With a word of joy, the sound of music fills the neighborhood."

[0058] Of course, in order to make the generated verses and blessings more in line with the expected requirements, in the present invention, the verses and blessings can be screened and filtered. That is, the generated verses and blessings are screened and filtered by the evaluation index to ensure that the screened and filtered verses and blessings are more in line with the expected requirements.

[0059] The evaluation indicators include coherence, rhythm and artistic conception.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for generating poems and blessings based on a large model, characterized in that: The following steps are involved: 1) Data preprocessing: Preprocess a large amount of poetry and blessing data to form a large-scale training data set; 2) Model pre-training: using a large pre-trained language model as a generative model, and pre-training the generative model with the large-scale training data set to obtain a pre-trained generative model; 3) Instruction fine-tuning: construct training data of instructions plus poems and blessings, and use this form of training data to fine-tune the pre-trained generation model to obtain a poem and blessing generation model; 4) Verses and blessings generation: The verses and blessings themes and instructions are not pre-processed for theme extraction and keyword expansion, but are directly input into the verses and blessings generation model to generate verses and blessings related to the verses and blessings themes.

2. The method for generating poems and blessings based on a large model according to claim 1 is characterized in that: The preprocessing in step 1) is to tokenize the verses and blessing data to segment the verses and blessing data into sequences of characters or words.

3. The method for generating poems and greetings based on a large model according to claim 1, characterized in that: In the step 1), the large amount of poetry and blessing data is greater than 10,000 poetry and blessing data and the large-scale training data set includes more than 10,000 training data.

4. The method for generating poems and greetings based on a large model according to claim 1, characterized in that: In step 2), the large pre-trained language model used is the GLM model. That is, generalized linear models.

5. The method for generating poems and blessings based on a large model according to claim 4 is characterized in that: In the step 2), the GLM model used is a GLM model with 10 billion parameters.

6. The method for generating verses and blessings based on a large model according to claim 5 is characterized in that: In the step 2), during the training process, the generation model is optimized through incremental pre-training to improve its generation quality and generalization ability.

7. The method for generating poems and blessings based on a large model according to any one of claims 1 to 6, characterized in that: Further including: 5) Screening and filtering of poems and blessings: The generated poems and blessings are screened and filtered through evaluation indicators to ensure that they meet the expected requirements.

8. The method for generating poems and greetings based on a large model according to claim 7 is characterized in that: The evaluation indicators include coherence, rhythm and artistic conception.

Images