A poetry generation and interpretation method based on a small parameter language model

Abstract

The present invention provides a poetry generation and interpretation method based on a small-parameter language model, relating to the field of machine learning technology. The method comprises the following steps: analyzing poetry parsing data using language instructions to obtain poetry training data; constructing a small-parameter language model using an encoder layer comprising an attention module and a feedforward network module; training the small-parameter language model using a multi-stage training method to obtain a trained small-parameter language model, thereby completing the construction of the small-parameter language model; wherein the small-parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results. The present invention solves the problems of poor adaptability of language models in Chinese environments and low generation quality.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a method for generating and interpreting poetry based on a small-parameter language model. Background Art

[0002] Large language models (LLMs) have recently become a focus of both academia and industry due to their ability to follow instructions and generate near-human responses. Models such as GPT-4 and LLAMA have demonstrated outstanding performance across a variety of natural language processing (NLP) tasks, highlighting their enormous potential for understanding and generating language. However, a significant limitation is that these models are primarily trained in English, and therefore exhibit certain limitations when processing Chinese contexts. They fail to fully understand the complex grammatical structure, culturally specific expressions, and semantic nuances of Chinese, resulting in reduced application effectiveness.

[0003] At the same time, large-scale language models tailored for the Chinese environment, such as Qwen and ChatGLM, have demonstrated good adaptability and accuracy in handling Chinese tasks, alleviating the acclimatization issues faced by cross-lingual models when directly applied to Chinese. However, their design as general-purpose models also introduces new challenges. These models often need to maintain a balance across a wide range of tasks, which inevitably limits their depth and accuracy in specific domains or specialized tasks. Despite their large scale, they struggle to achieve the same level of expertise as domain-specific models across all domains, especially in scenarios requiring deep domain knowledge or highly specialized language expressions. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a poetry generation and interpretation method based on a small-parameter language model, which solves the problems of poor adaptability of the language model in the Chinese environment and low generation quality.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for generating and interpreting poetry based on a small-parameter language model, comprising:

[0006] S1: Analyze poetry parsing data using language instructions to obtain poetry training data;

[0007] S2: Use the encoder layer containing the attention module and the feedforward network module to build a small parameter language model;

[0008] S3: Use a multi-stage training method to train the small-parameter language model to obtain a trained small-parameter language model, and complete the construction of the small-parameter language model; wherein the small-parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results.

[0009] Furthermore, the S1 includes:

[0010] Using the text generation model, we can generate poetry and obtain poetry analysis data;

[0011] Analyze the poetry analysis data using language instructions to obtain theme poetry data; wherein the theme poetry data includes theme, keywords, content and emotion;

[0012] The theme poetry data is normalized to obtain poetry training data:

[0013]

[0014]

[0015] Among them, x 诗词 Represents poetry training data, x represents theme poetry data, Mean represents average value operation, ò represents preset constant, N represents the total number of theme poetry data, and i1 represents the sequence number of theme poetry data.

[0016] Furthermore, the small parameter language model includes a plurality of encoder layers, wherein the encoder layer includes:

[0017] An attention module, including a position encoding algorithm, is used to calculate the relevance of each element in the poetry training data through a multi-head attention mechanism to obtain an attention feature vector;

[0018] The feedforward network module includes an input unit, a hidden unit and an output unit, and is used to map the attention feature vector to a high-dimensional space through linear transformation and nonlinear activation to obtain high-dimensional spatial feature information.

[0019] Furthermore, the expression of the attention feature vector is:

[0020]

[0021] X q,k =X′ q,k ·F cis ;

[0022]

[0023] Among them, attn(Q,K,V) represents the attention feature vector, Q represents the query, V represents the value, softmax represents the softmax activation function, and d k Indicates the dimension of the key, X q,k represents the input before embedding, X' q,k Represents Q and K, F converted to complex domain cisrepresents the transformation requirement of position embedding, j represents the time step, i f represents the index of the frequency component, d represents the vector dimension, and θ represents the scaling factor;

[0024] The expression of the nonlinear activation is:

[0025]

[0026] Among them, FFN SwiGLU Represents the nonlinear activation result, x a Represents the data of the input unit, W1, W2 and W V Represents the weight of the fully connected layer, SiLu represents the Sigmoid linear unit activation function, Represents multiplication operation;

[0027] The expression of the small parameter language model is:

[0028] y=layern(...layer2(layer1(xpoetry;θ1);θ2);θn);

[0029] Among them, layer n represents the nth coding layer, layer2 represents the second coding layer, layer1 represents the first coding layer, and x 诗词 represents poetry training data, θ1 represents the parameters of the first encoding layer, θ2 represents the parameters of the second encoding layer, θ n Represents the parameters of the nth coding layer.

[0030] Furthermore, the S3 includes:

[0031] Using the pre-training data set, performing a first pre-training on the small-parameter language model to obtain a small-parameter language model after the first pre-training;

[0032] Using the poetry training data, the small parameter language model after the first pre-training is pre-trained for a second time to obtain the small parameter language model after the second pre-training; wherein the loss function expression of the multi-stage training is:

[0033]

[0034] Among them, L θ Represents the loss function result of the second pre-training, P θ Represents the predicted probability of the newly generated token, y i Represents the predicted probability of the i-th newly generated token, y 输出 Indicates that the target sequence is located at y i All elements before, P represents the previous information, θ represents the trainable weight, and M represents the total length of the sequence;

[0035] The small-parameter language model after the second pre-training is fine-tuned using a full-parameter fine-tuning strategy to obtain a trained small-parameter language model.

[0036] The beneficial effects of the present invention include: using a small-parameter language model obtained through multi-stage training to process poetry training data, obtaining poetry generation and interpretation results. By refining and optimizing the model at different stages, the model's generation quality and comprehension capabilities are significantly improved, enhancing the model's professional level and creativity in the field of poetry creation, as well as its performance in diverse task environments. The resulting small-parameter language model has a small number of parameters, low hardware requirements, and high training efficiency, while also performing well in poetry generation and comprehension. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0038] Figure 1 This is an exemplary flowchart of a method for generating and interpreting poetry based on a small-parameter language model according to some embodiments of this specification. DETAILED DESCRIPTION

[0039] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0040] Example

[0041] Figure 1 This is an exemplary flow chart of a method for generating and interpreting poetry based on a small parameter language model according to some embodiments of this specification. Figure 1 As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0042] S1: Use language instructions to analyze poetry parsing data to obtain poetry training data.

[0043] Poetry parsing data is text data containing poetry and corresponding instructions. For example, poetry parsing data can include ancient classical Chinese texts, poetry works, and other literary works.

[0044] In some embodiments, the processor can generate poetry parsing data based on specific instructions through text generation models such as GPT3.5 and GPT4.0.

[0045] Poetry training data is text data used to train small-parameter language models.

[0046] In some embodiments, the processor can implement S1 based on the following steps: using a text generation model to generate poetry to obtain poetry parsing data; using language instructions to analyze the poetry parsing data to obtain theme poetry data; normalizing the theme poetry data to obtain poetry training data.

[0047] Theme poetry data is poetry data with multiple themes, such as themes, keywords, content, and emotions.

[0048] In some embodiments, the processor may use language instructions to analyze the poetry parsing data to obtain theme poetry data.

[0049] The language instruction is an instruction for extracting a topic from the poetry parsing data. For example, the specific types of the language instruction can be shown in Table 1.

[0050] Table 1

[0051]

[0052]

[0053] In some embodiments, the expression of poetry training data can be:

[0054]

[0055]

[0056] Among them, x 诗词 Represents poetry training data, x represents theme poetry data, Mean represents average value operation, ò represents preset constant, N represents the total number of theme poetry data, and i1 represents the sequence number of theme poetry data.

[0057] S2: Build a small-parameter language model using the encoder layer consisting of an attention module and a feed-forward network module.

[0058] Small-parameter language models are used to analyze poetry training data to generate and interpret poetry. There are many types of small-parameter language models. For example, they can include feedforward neural network models.

[0059] In some embodiments, the input of the small-parameter language model may be poetry training data, and the output of the small-parameter language model may be poetry generation and interpretation results.

[0060] In some embodiments, the structure of the small parameter language model is as follows:

[0061] The low-parameter language model consists of multiple encoder layers, each of which includes an attention module and a feedforward network module. The output of the attention module serves as the input to the feedforward network module, which in turn serves as the input to the next encoder layer. The output of the last encoder layer serves as the final output of the low-parameter language model.

[0062] In some embodiments, the expression of the small parameter language model may be:

[0063] y=layer n (…layer2(layer1(x 诗词 ;θ1);θ2);θ n );

[0064] Among them, layer n represents the nth coding layer, layer2 represents the second coding layer, layer1 represents the first coding layer, and x 诗词 represents poetry training data, θ1 represents the parameters of the first encoding layer, θ2 represents the parameters of the second encoding layer, θ n Represents the parameters of the nth coding layer.

[0065] The attention module is used to calculate the correlation of each element in the poetry training data through a multi-head attention mechanism to obtain an attention feature vector. The input of the attention module may include the poetry training data, and the output may include the attention feature vector.

[0066] The attention module may include a position encoding algorithm.

[0067] The attention feature vector is a feature vector with different weights for different topic dimensions.

[0068] In some embodiments, the expression of the attention feature vector can be:

[0069]

[0070] X q,k =X′ q,k ·F cis ;

[0071]

[0072] Among them, attn(Q,K,V) represents the attention feature vector, Q represents the query, V represents the value, softmax represents the softmax activation function, and d k Indicates the dimension of the key, X q,k represents the input before embedding, X' q,k Represents Q and K, F converted to complex domain cis represents the transformation requirement of position embedding, j represents the time step, i f represents the index of the frequency component, d represents the vector dimension, and θ represents the scaling factor.

[0073] The feedforward network module is used to map the attention feature vector to a high-dimensional space through linear transformation and nonlinear activation to obtain high-dimensional spatial feature information. The input of the feedforward network module may include the attention feature vector, and the output may include high-dimensional spatial feature information.

[0074] The feedforward network module includes input units, hidden units, and output units. The output of the input unit serves as the input of the hidden unit, the output of the hidden unit serves as the input of the output unit, and the output of the output unit serves as the final output of the feedforward network module.

[0075] High-dimensional spatial feature information is attention feature information with higher dimensions.

[0076] In some embodiments, the output unit may process the output data of the hidden unit using a nonlinear activation method to obtain high-dimensional spatial feature information.

[0077] In some embodiments, the processor may use the high-dimensional spatial feature information output by the last encoder layer as the result of poetry generation and interpretation.

[0078] In some embodiments, the expression of nonlinear activation may be:

[0079]

[0080] Among them, FFN SwiGLU Represents the nonlinear activation result, x a Represents the data of the input unit, W1, W2 and W V Represents the weight of the fully connected layer, SiLu represents the Sigmoid linear unit activation function, Represents a multiplication operation.

[0081] S3: Use a multi-stage training method to train the small-parameter language model to obtain a trained small-parameter language model, and complete the construction of the small-parameter language model; wherein the small-parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results.

[0082] The poetry generation and interpretation results are the poetry generation text data based on the interpretation instruction requirements.

[0083] In some embodiments, the processor can use a small-parameter language model to analyze poetry training data to obtain poetry generation and interpretation results. For example, the processor can input the poetry training data into the attention module of the encoder layer to extract attention features and obtain attention feature vectors; the attention feature vectors can be input into the feedforward network module to obtain high-dimensional spatial feature information, and the high-dimensional spatial feature information output by the last encoder layer can be used as the poetry generation and interpretation results.

[0084] The S3 includes:

[0085] Using the pre-training data set, the small-parameter language model is pre-trained for the first time to obtain a small-parameter language model after the first pre-training; using the poetry training data, the small-parameter language model after the first pre-training is pre-trained for the second time to obtain a small-parameter language model after the second pre-training; using the full-parameter fine-tuning strategy, the small-parameter language model after the second pre-training is fine-tuned to obtain a trained small-parameter language model.

[0086] In some embodiments, the loss function expression for multi-stage training can be:

[0087]

[0088] Among them, L θ Represents the loss function result of the second pre-training, P θ Represents the predicted probability of the newly generated token, y i Represents the predicted probability of the i-th newly generated token, y 输出 Indicates that the target sequence is located at y i All the elements before, P represents the previous information, θ represents the trainable weight, and M represents the total length of the sequence.

[0089] Pre-training datasets are text training datasets with rich linguistic phenomena and cultural connotations. For example, pre-training datasets can include the CLUECorpusSmall dataset and the ChinesePoetry dataset.

[0090] In some embodiments, the first pre-training process may include inputting the pre-training data set into the small-parameter language model for training, wherein the effective batch size is 16, the maximum learning rate is 1×10^-4, the learning rate warm-up period is 5000 steps, the training process is planned to be 34398 steps, and 3 complete iterations are performed on the selected data set to obtain the small-parameter language model after the first pre-training.

[0091] In some embodiments, the second pre-training process may include inputting the poetry training data into the small-parameter language model after the first pre-training for training, wherein the effective batch size is 16, the maximum learning rate is 5×10^-5, and 10 rounds of iterations are planned to obtain the small-parameter language model after the second pre-training.

[0092] In some embodiments, the processor may utilize a full-parameter fine-tuning strategy to fine-tune the small-parameter language model after the second pre-training to obtain a trained small-parameter language model. Specifically, the processor may first perform preliminary fine-tuning on the poetry training data, then gradually introduce other task data for comprehensive fine-tuning, and finally fine-tune all the data together to obtain a trained small-parameter language model.

[0093] In some embodiments of this specification, a small-parameter language model, obtained through multi-stage training, is used to process poetry training data to generate and interpret poetry. By refining and optimizing the model at different stages, the model's generation quality and comprehension capabilities are significantly improved, enhancing the model's professionalism, creativity, and performance in diverse task environments in the field of poetry creation. The resulting small-parameter language model features a small number of parameters, low hardware requirements, and high training efficiency, while also excelling in both poetry generation and comprehension.

Claims

1. A method for generating and interpreting poetry based on a small-parameter language model, characterized in that: include: S1: Analyze poetry parsing data using language instructions to obtain poetry training data; S2: Use the encoder layer containing the attention module and the feedforward network module to build a small parameter language model; The small parameter language model includes a plurality of encoder layers, wherein the encoder layer includes: An attention module, including a position encoding algorithm, is used to calculate the relevance of each element in the poetry training data through a multi-head attention mechanism to obtain an attention feature vector; The feedforward network module includes an input unit, a hidden unit, and an output unit, and is used to map the attention feature vector to a high-dimensional space through linear transformation and nonlinear activation to obtain high-dimensional spatial feature information; the expression of the attention feature vector is: ； ; ； in, represents the attention feature vector, Indicates a query, Represents a value, represents the softmax activation function, represents the dimension of the key, represents the input before embedding, Indicates conversion to plural domain and , represents the transformation requirements for position embedding, represents the time step, represents the index of the frequency component, represents the vector dimension, represents the scaling factor; The expression of the nonlinear activation is: ； in, represents the nonlinear activation result, Represents the data of the input unit, 、 and represents the weight of the fully connected layer, represents the Sigmoid linear unit activation function, Represents multiplication operation; The expression of the small parameter language model is: ； in, Indicates the Coding layer, represents the second coding layer, represents the first coding layer, represents poetry training data, represents the parameters of the first coding layer, represents the parameters of the second coding layer, Indicates the Parameters of the coding layer; S3: Use a multi-stage training method to train the small-parameter language model to obtain a trained small-parameter language model, and complete the construction of the small-parameter language model; wherein the small-parameter language model is used to analyze the poetry training data to obtain poetry generation and interpretation results.

2. The method for constructing a small parameter language model for poetry generation and interpretation according to claim 1, characterized in that: Said S1 comprises: Using the text generation model, we can generate poetry and obtain poetry analysis data; Analyze the poetry analysis data using language instructions to obtain theme poetry data; wherein the theme poetry data includes theme, keywords, content and emotion; The theme poetry data is normalized to obtain poetry training data: ； ； in, represents poetry training data, Indicates the theme poem data, represents the average operation, Represents a preset constant, Indicates the total number of theme poetry data, Indicates the serial number of the theme poem data.

3. The method for constructing a small parameter language model for poetry generation and interpretation according to claim 1, characterized in that: The S3 includes: Using the pre-training data set, performing a first pre-training on the small-parameter language model to obtain a small-parameter language model after the first pre-training; Using the poetry training data, the small parameter language model after the first pre-training is pre-trained for a second time to obtain the small parameter language model after the second pre-training; wherein the loss function expression of the multi-stage training is: ； in, Represents the loss function result of the second pre-training, Represents the predicted probability of the newly generated token, Indicates the The predicted probability of a newly generated token, Indicates that the target sequence is located at All the elements before Indicates the previous information. represents the trainable weights, Indicates the total length of the sequence; The small-parameter language model after the second pre-training is fine-tuned using a full-parameter fine-tuning strategy to obtain a trained small-parameter language model.

Images