Poem generation and interpretation method based on small parameter language model

Abstract

The invention provides a poem generation and interpretation method based on a small parameter language model, and relates to the technical field of machine learning, and the method comprises the steps: analyzing poem analysis data through a language instruction, and obtaining poem training data; constructing a small parameter language model by using an encoder layer comprising an attention module and a feedforward network module; training the small-parameter language model by using a multi-stage training mode to obtain a trained small-parameter language model, and completing the construction of the small-parameter language model; wherein the small parameter language model is used for analyzing the poem training data to obtain a poem generation and interpretation result. According to the method, the problems of poor adaptive capacity and low generation quality of a language model in a Chinese environment are solved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method for generating and interpreting poetry based on a small parameter language model. Background Art

[0002] Recently, Large Language Models (LLMs) have become the focus of academia and industry due to their ability to follow instructions and generate human-like responses. Models such as GPT-4 and LLAMA have demonstrated excellent performance in various natural language processing (NLP) tasks, highlighting their great potential in understanding and generating language. However, a limitation that cannot be ignored is that these models are mainly trained based on English, and therefore show certain limitations when processing Chinese contexts. They fail to fully understand the complex grammatical structure, culturally specific expressions, and semantic nuances of Chinese, resulting in reduced application effects.

[0003] At the same time, large language models customized for the Chinese environment, such as Qwen and ChatGLM, have shown good adaptability and accuracy in processing Chinese tasks, alleviating the problem of cross-language models not being able to adapt to Chinese when directly applied to Chinese. However, their design positioning as general models has also led to new challenges. Such models often need to maintain a balance across a wide range of tasks, which inevitably limits their depth and accuracy in specific fields or professional tasks. Despite their large scale, they are difficult to achieve a professional level comparable to domain-specific models in all fields, especially in scenarios that require deep domain knowledge or highly specialized language expressions. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a poetry generation and interpretation method based on a small-parameter language model, which solves the problems of poor adaptability of the language model in the Chinese environment and low generation quality.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a poetry generation and interpretation method based on a small parameter language model, comprising:

[0006] S1: Analyze poetry analysis data using language instructions to obtain poetry training data;

[0007] S2: Use the encoder layer containing the attention module and the feed-forward network module to build a small parameter language model;

[0008] S3: Train the small parameter language model using a multi-stage training method to obtain a trained small parameter language model, and complete the construction of the small parameter language model; wherein the small parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results.

[0009] Further, the S1 includes:

[0010] Use the text generation model to generate poems and obtain poetry analysis data;

[0011] Analyze the poetry analysis data using language instructions to obtain theme poetry data; wherein the theme poetry data includes theme, keywords, content and emotion;

[0012] The theme poetry data is normalized to obtain poetry training data:

[0013]

[0014]

[0015] Among them, x 诗词 represents poetry training data, x represents the theme poetry data, Mean represents the average value operation, ò represents the preset constant, N represents the total number of theme poetry data, i 1 Indicates the serial number of the theme poem data.

[0016] Further, the small parameter language model includes a plurality of encoder layers, wherein the encoder layer includes:

[0017] An attention module, including a position encoding algorithm, for calculating the relevance of each element in the poetry training data through a multi-head attention mechanism to obtain an attention feature vector;

[0018] The feedforward network module includes an input unit, a hidden unit and an output unit, and is used to map the attention feature vector to a high-dimensional space through linear transformation and nonlinear activation to obtain high-dimensional space feature information.

[0019] Furthermore, the expression of the attention feature vector is:

[0020]

[0021] X q,k =X′ q,k ·F cis ;

[0022]

[0023] Where attn(Q,K,V) represents the attention feature vector, Q represents the query, V represents the value, softmax represents the softmax activation function, and d k Indicates the dimension of the key, X q,k represents the input before embedding, X' q,k Represents Q and K, F converted to complex domaincis represents the transformation requirement of position embedding, j represents the time step, i f represents the index of the frequency component, d represents the vector dimension, and θ represents the scaling factor;

[0024] The expression of the nonlinear activation is:

[0025]

[0026] Among them, FFN SwiGLU Represents the nonlinear activation result, x a Represents the data of the input unit, W 1 , W 2 and W V represents the weight of the fully connected layer, SiLu represents the Sigmoid linear unit activation function, Represents multiplication operation;

[0027] The expression of the small parameter language model is:

[0028] y=layern(…layer 2 (layer 1 (x poetry; θ 1 );θ 2 );θn);

[0029] Among them, layer n Indicates the nth coding layer, layer 2 Indicates the second coding layer, layer 1 represents the first encoding layer, x 诗词 represents poetry training data, θ 1 represents the parameters of the first coding layer, θ 2 represents the parameters of the second coding layer, θ n Represents the parameters of the nth coding layer.

[0030] Further, the S3 includes:

[0031] Using the pre-training data set, performing a first pre-training on the small-parameter language model to obtain a small-parameter language model after the first pre-training;

[0032] Using the poetry training data, the small parameter language model after the first pre-training is pre-trained for the second time to obtain the small parameter language model after the second pre-training; wherein the loss function expression of the multi-stage training is:

[0033]

[0034] Among them, L θ Represents the loss function result of the second pre-training, Pθ Represents the predicted probability of the newly generated token, y i represents the predicted probability of the i-th newly generated token, y 输出 Indicates that the target sequence is located at y i All the elements before, P represents the previous information, θ represents the trainable weight, and M represents the total length of the sequence;

[0035] The small-parameter language model after the second pre-training is fine-tuned using a full-parameter fine-tuning strategy to obtain a trained small-parameter language model.

[0036] The beneficial effects of the present invention are: using a small parameter language model obtained through multi-stage training to process poetry training data, the poetry generation and interpretation results are obtained. By refining and optimizing the model at different stages, the generation quality and understanding ability of the model are significantly improved, and the professional level, creative ability and performance of the model in the field of poetry creation in a variety of task environments are improved; the small parameter language model obtained has a small number of parameters, low hardware requirements, high training efficiency, and excellent performance in poetry generation and understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0038] Figure 1 It is an exemplary flow chart of a method for generating and interpreting poetry based on a small parameter language model according to some embodiments of this specification. DETAILED DESCRIPTION

[0039] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0040] Example

[0041] Figure 1 This is an exemplary flow chart of a method for generating and interpreting poetry based on a small parameter language model according to some embodiments of this specification. Figure 1 As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0042] S1: Use language instructions to analyze poetry parsing data to obtain poetry training data.

[0043] Poetry analysis data is text data containing poetry and corresponding instructions. For example, poetry analysis data may include ancient classical Chinese, poetry works and other literary works.

[0044] In some embodiments, the processor can generate poetry parsing data based on specific instructions through text generation models such as GPT3.5 and GPT4.0.

[0045] Poetry training data is text data used to train a small-parameter language model.

[0046] In some embodiments, the processor can implement S1 based on the following steps: using a text generation model to generate poetry to obtain poetry parsing data; using language instructions to analyze the poetry parsing data to obtain theme poetry data; normalizing the theme poetry data to obtain poetry training data.

[0047] Theme poetry data is poetry data with multiple theme contents. For example, theme poetry data includes theme, keywords, content and emotion.

[0048] In some embodiments, the processor can use language instructions to analyze the poetry parsing data to obtain theme poetry data.

[0049] The language instruction is an instruction for extracting a topic from the poetry parsing data. For example, the specific types of the language instruction may be as shown in Table 1.

[0050] Table 1

[0051]

[0052]

[0053] In some embodiments, the expression of poetry training data may be:

[0054]

[0055]

[0056] Among them, x 诗词 represents poetry training data, x represents the theme poetry data, Mean represents the average value operation, ò represents the preset constant, N represents the total number of theme poetry data, i 1 Indicates the data serial number of the theme poem.

[0057] S2: Construct a small parameter language model using the encoder layer consisting of an attention module and a feed-forward network module.

[0058] The small parameter language model is used to analyze the poetry training data to obtain the poetry generation and interpretation results. There can be many types of small parameter language models. For example, the type of small parameter language model can include a feedforward neural network model, etc.

[0059] In some embodiments, the input of the small parameter language model may be poetry training data, and the output of the small parameter language model may be poetry generation and interpretation results.

[0060] In some embodiments, the structure of the small parameter language model is as follows:

[0061] The small parameter language model includes multiple encoder layers, wherein the encoder layer includes an attention module and a feedforward network module. The output of the attention module is used as the input of the feedforward network module, the output of the feedforward network module is used as the input of the next encoder layer, and the output of the last encoder layer is used as the final output of the small parameter language model.

[0062] In some embodiments, the expression of the small parameter language model may be:

[0063] y=layer n (…layer 2 (layer 1 (x 诗词 θ 1 );θ 2 );θ n );

[0064] Among them, layer n Indicates the nth coding layer, layer 2 Indicates the second coding layer, layer 1 represents the first encoding layer, x 诗词 represents poetry training data, θ 1 represents the parameters of the first coding layer, θ 2 represents the parameters of the second coding layer, θ n Represents the parameters of the nth coding layer.

[0065] The attention module is used to calculate the correlation of each element in the poetry training data through a multi-head attention mechanism to obtain an attention feature vector. The input of the attention module may include the poetry training data, and the output may include the attention feature vector.

[0066] The attention module may include a position encoding algorithm.

[0067] The attention feature vector is a feature vector with different weights for different topic dimensions.

[0068] In some embodiments, the expression of the attention feature vector may be:

[0069]

[0070] X q,k =X′ q,k ·F cis ;

[0071]

[0072] Where attn(Q,K,V) represents the attention feature vector, Q represents the query, V represents the value, softmax represents the softmax activation function, and d k Indicates the dimension of the key, X q,k represents the input before embedding, X' q,k Represents Q and K, F converted to complex domain cis represents the transformation requirement of position embedding, j represents the time step, i f represents the index of the frequency component, d represents the vector dimension, and θ represents the scaling factor.

[0073] The feedforward network module is used to map the attention feature vector to a high-dimensional space through linear transformation and nonlinear activation to obtain high-dimensional spatial feature information. The input of the feedforward network module may include the attention feature vector, and the output may include high-dimensional spatial feature information.

[0074] The feedforward network module includes an input unit, a hidden unit and an output unit. The output of the input unit is used as the input of the hidden unit, the output of the hidden unit is used as the input of the output unit, and the output of the output unit is used as the final output of the feedforward network module.

[0075] High-dimensional spatial feature information is attention feature information with higher dimensions.

[0076] In some embodiments, the output unit may process the output data of the hidden unit in a nonlinear activation manner to obtain high-dimensional spatial feature information.

[0077] In some embodiments, the processor may use the high-dimensional spatial feature information output by the last encoder layer as the result of poetry generation and interpretation.

[0078] In some embodiments, the expression of nonlinear activation may be:

[0079]

[0080] Among them, FFN SwiGLU Represents the nonlinear activation result, x a Represents the data of the input unit, W 1 , W 2 and W Vrepresents the weight of the fully connected layer, SiLu represents the Sigmoid linear unit activation function, Represents the multiplication operation.

[0081] S3: Train the small parameter language model using a multi-stage training method to obtain a trained small parameter language model, and complete the construction of the small parameter language model; wherein the small parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results.

[0082] The poetry generation and interpretation results are the poetry generation text data based on the interpretation instruction requirements.

[0083] In some embodiments, the processor can use a small parameter language model to analyze the poetry training data to obtain poetry generation and interpretation results. For example, the processor can input the poetry training data into the attention module of the encoder layer, extract the attention features, and obtain the attention feature vector; input the attention feature vector into the feedforward network module to obtain high-dimensional space feature information, and use the high-dimensional space feature information output by the last encoder layer as the poetry generation and interpretation result.

[0084] The S3 includes:

[0085] The small parameter language model is pre-trained for the first time using the pre-training data set to obtain a small parameter language model after the first pre-training; the small parameter language model after the first pre-training is pre-trained for the second time using the poetry training data to obtain a small parameter language model after the second pre-training; the small parameter language model after the second pre-training is fine-tuned using the full parameter fine-tuning strategy to obtain a trained small parameter language model.

[0086] In some embodiments, the loss function expression of multi-stage training can be:

[0087]

[0088] Among them, L θ Represents the loss function result of the second pre-training, P θ Represents the predicted probability of the newly generated token, y i represents the predicted probability of the i-th newly generated token, y 输出 Indicates that the target sequence is located at y i All the elements before, P represents the previous information, θ represents the trainable weight, and M represents the total length of the sequence.

[0089] The pre-training dataset is a text training dataset with rich language phenomena and cultural connotations. For example, the pre-training dataset may include the CLUECorpusSmall dataset and the ChinesePoetry dataset.

[0090] In some embodiments, the first pre-training process may include inputting the pre-training data set into a small parameter language model for training, wherein the effective batch size is 16, the maximum learning rate is 1×10^-4, the learning rate warm-up period is 5000 steps, the training process is planned to be 34398 steps, and 3 complete iterations are performed on the selected data set to obtain the small parameter language model after the first pre-training.

[0091] In some embodiments, the second pre-training process may include inputting the poetry training data into the small-parameter language model after the first pre-training for training, wherein the effective batch size is 16, the maximum learning rate is 5×10^-5, 10 rounds of iterations are planned, and the small-parameter language model after the second pre-training is obtained.

[0092] In some embodiments, the processor can use a full parameter fine-tuning strategy to fine-tune the small parameter language model after the second pre-training to obtain a trained small parameter language model. Specifically, the processor can first perform preliminary fine-tuning on the poetry training data, then gradually introduce other task data for comprehensive fine-tuning, and finally fine-tune all the data to obtain a trained small parameter language model.

[0093] In some embodiments of this specification, the poetry training data is processed using a small parameter language model obtained through multi-stage training to obtain poetry generation and interpretation results. By refining and optimizing the model at different stages, the generation quality and understanding ability of the model are significantly improved, and the model's professional level, creative ability and performance in a variety of task environments in the field of poetry creation are improved; the small parameter language model obtained has a small number of parameters, low hardware requirements, high training efficiency, and excellent performance in poetry generation and understanding.

Claims

1. A poetry generation and interpretation method based on a small parameter language model, characterized in that: include: S1: Analyze poetry analysis data using language instructions to obtain poetry training data; S2: Use the encoder layer containing the attention module and the feed-forward network module to build a small parameter language model; S3: Train the small parameter language model using a multi-stage training method to obtain a trained small parameter language model, and complete the construction of the small parameter language model; wherein the small parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results.

2. The method for constructing a small parameter language model for poetry generation and interpretation according to claim 1, characterized in that: The S1 includes: Use the text generation model to generate poems and obtain poetry analysis data; Analyze the poetry analysis data using language instructions to obtain theme poetry data; wherein the theme poetry data includes theme, keywords, content and emotion; The theme poetry data is normalized to obtain poetry training data: Among them, x 诗词 Represents poetry training data, x represents theme poetry data, Mean represents the average value operation, ò represents the preset constant, N represents the total number of theme poetry data, and i1 represents the sequence number of theme poetry data.

3. The method for constructing a small parameter language model for poetry generation and interpretation according to claim 1, characterized in that: The small parameter language model comprises a plurality of encoder layers, wherein the encoder layers comprise: An attention module, including a position encoding algorithm, for calculating the relevance of each element in the poetry training data through a multi-head attention mechanism to obtain an attention feature vector; The feedforward network module includes an input unit, a hidden unit and an output unit, and is used to map the attention feature vector to a high-dimensional space through linear transformation and nonlinear activation to obtain high-dimensional space feature information.

4. The method for constructing a small parameter language model for poetry generation and interpretation according to claim 3 is characterized in that: The expression of the attention feature vector is: X q,k =X' q,k ·F cis ; Where attn(Q,K,V) represents the attention feature vector, Q represents the query, V represents the value, softmax represents the softmax activation function, and d k Indicates the dimension of the key, X q,k represents the input before embedding, X' q,k Represents Q and K, F converted to complex domain cis represents the transformation requirement of position embedding, j represents the time step, i f represents the index of the frequency component, d represents the vector dimension, and θ represents the scaling factor; The expression of the nonlinear activation is: Among them, FFN SwiGLU Represents the nonlinear activation result, x a Represents the data of the input unit, W1, W2 and W V represents the weight of the fully connected layer, SiLu represents the Sigmoid linear unit activation function, Represents multiplication operation; The expression of the small parameter language model is: y=layern(...layer2(layer1(x poem; θ1); θ2); θn); Among them, layer n represents the nth coding layer, layer2 represents the second coding layer, layer1 represents the first coding layer, and x 诗词 represents poetry training data, θ1 represents the parameters of the first encoding layer, θ2 represents the parameters of the second encoding layer, and θ n Represents the parameters of the nth coding layer.

5. The method for constructing a small parameter language model for poetry generation and interpretation according to claim 1, characterized in that: The S3 includes: Using the pre-training data set, performing a first pre-training on the small-parameter language model to obtain a small-parameter language model after the first pre-training; Using the poetry training data, the small parameter language model after the first pre-training is pre-trained for the second time to obtain the small parameter language model after the second pre-training; wherein the loss function expression of the multi-stage training is: Among them, L θ Represents the loss function result of the second pre-training, P θ Represents the predicted probability of the newly generated token, y i represents the predicted probability of the i-th newly generated token, y 输出 Indicates that the target sequence is located at y i All the elements before, P represents the previous information, θ represents the trainable weight, and M represents the total length of the sequence; The small-parameter language model after the second pre-training is fine-tuned using a full-parameter fine-tuning strategy to obtain a trained small-parameter language model.

Images