Text generation method, system and equipment based on large language model and medium
By introducing dynamic adjustment factor and error -correction discipline in the large language model, combined with Chinese expression rules, the text generation process is optimized, solving the problem of limited learning capabilities and data dependence in the long sequence processing of the Transformer class, and realized it. High accuracy and efficiency.
Patent Information
- Application Number
- CN202411130347.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2025-01-21
AI Technical Summary
When processing long sequences, the Transformer class model has limited learning ability, and pre-training and fine-tuning require a large amount of data and computing resources.
Dynamic adjustment factors and error correction filters are used, combined with Chinese expression rules, to optimize the text generation process of the large language model, and dynamically adjust the probability of the target vocabulary until it meets the regulations.
It improves the accuracy and efficiency of text generation, reduces the dependence on a large amount of test data, and enhances the model's learning ability in long sequence processing.
Smart Images
Figure CN119337870A_ABST
Abstract
Description
Background Art
[0001] The present application belongs to the technical field of automatic generation of Chinese text, and specifically relates to a text generation method, system, device and medium based on a large language model. Technical Background
[0003] Transformer models use self-attention mechanisms to achieve encoding and representation learning of input sequences. Compared with traditional recurrent neural networks (RNN) or convolutional neural networks (CNN), they have higher parallelism and computational efficiency. However, they usually require a large amount of data and computing resources for pre-training and fine-tuning. Especially in natural language processing tasks in some niche areas, more professional data and domain knowledge are often required to improve the generalization ability of the model. However, for longer sequences, the learning ability of the model is still limited. This is because the self-attention mechanism captures the dependency between different positions based on local contextual information. As the sequence length increases, the performance of the model will be affected to a certain extent. Summary of the invention
[0004] Based on the above problems, this application uses dynamic adjustment factors to alleviate the problem that pre-training and fine-tuning require a large amount of test data, and uses Chinese expression rules to capture contextual information to improve accuracy. Its technical solution is:
[0005] A text generation method based on a large language model comprises the following steps:
[0006] S1. Preprocess the keywords input by the user and convert the input keywords into corresponding word embedding vectors;
[0007] S2. The large language model is based on the Transformer architecture and gives a set of target vocabulary T (t1, t2, ... t n ) and probability P(p1,p2,…p n ), and add an error correction filter after the self-attention layer of the Transformer architecture; set a dynamic adjustment factor in the error correction filter;
[0008] S3 dynamically adjusts the target vocabulary probability; repeats steps S2-S3 until the current target vocabulary meets the requirements;
[0009] S4. Output the result according to the preset template.
[0010] Preferably, in step S2, the large language model selects a large language model encoding algorithm based on the Transformer architecture for sample training; based on a natural language vectorization algorithm, the keywords are analyzed and converted into corresponding word embedding vectors;
[0011] The Transformer architecture includes an encoder and a decoder; the encoder and the decoder include an attention layer and a feedforward neural network layer; the decoding process of the large language model includes the following steps:
[0012] The decoder first applies a self-attention mechanism to the currently generated output sequence; calculates the dependency between the previous and next words;
[0013] The decoder interacts the state of the current decoding position with all hidden states generated by the encoder, and obtains relevant context information of the text sentence based on the encoder-decoder attention layer;
[0014] The decoder passes the combined vector to the feed-forward neural network layer;
[0015] The attention layer and the feedforward neural network layer add residual connections to retain the original input information; stabilize the training process based on layer normalization and optimize the gradient propagation of the text;
[0016] The decoder will go through a linear layer and generate a probability distribution vector based on the loss function, which generates the next target word and its probability.
[0017] Preferably, in step S3,
[0018] S31. Determine the current target vocabulary based on the existing text information and expression habits i The target word with the largest probability is selected. If the target word's part of speech matches the expression habit, no adjustment is required and the target word is used directly. If the target word's part of speech does not match the expression habit, the next target word is selected and step S31 is repeated until the target word's part of speech matches the expression habit, and its probability is adjusted using a dynamic adjustment factor, the formula is as follows:
[0019] p′ i =p i +(p max -ln2 r );
[0020] In the formula, p i is the corresponding probability in the target vocabulary, and r is a dynamic adjustment factor.
[0021] S32. Change p′ iFeedback is given to the large language model to re-output a new round of target vocabulary sets and their probabilities.
[0022] Preferably, in step S31, the large language model sets the probability of the target words that were not selected in the previous round to zero, determines that the current target word is unavailable, and performs pruning.
[0023] Preferably, in step S2, the error correction filter will check whether the replaced text complies with grammatical and semantic rules, and perform further error correction and correction as needed: if repeated words, ungrammatical sentence structures or obvious semantic errors appear in the replaced text, the error correction result filter will correct these problems.
[0024] A text generation device based on a large language model, comprising:
[0025] The acquisition module is used to obtain the word embedding vector of the keyword;
[0026] Extraction module, used to extract features through transformer, target vocabulary T(t1,t2,…t n ) and probability P(p1,p2,…p n );
[0027] Judgment module, judge the current target vocabulary t i Does it conform to current expression habits?
[0028] A generation module outputs a result according to a preset template. An electronic device includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the text generation method of the present application.
[0029] A computer-readable medium stores a computer program, wherein the computer program implements the text generation method of the present application when executed by a processor.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] This application utilizes dynamic adjustment factors to alleviate the problem that pre-training and fine-tuning require a large amount of test data, and utilizes language expression rules to capture contextual information to improve accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 This is the flow chart of this application. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0034] A text generation method based on a large language model comprises the following steps:
[0035] S1. Preprocess the keywords input by the user and convert the input keywords into corresponding word embedding vectors;
[0036] When using the model for prediction, the input text data needs to be preprocessed first. This preprocessing process mainly includes several steps such as word segmentation, part-of-speech tagging, dictionary construction, data cleaning and denoising, and text vectorization. Among them, word segmentation uses commonly used word segmentation tools such as jieba to divide the text into meaningful words for subsequent processing; part-of-speech tagging can use tagging tools such as Stanford Part-of-Speech Tagger to determine the part of speech for each word, such as noun, verb, adjective, etc.; dictionary construction is to encode each word based on the results of word segmentation and part-of-speech tagging, map each word to an integer value, or map each word to a vector representation; text vectorization uses some common methods, such as the bag-of-words model to convert the text data after word segmentation into a word vector representation, and obtain the word vector U=(u1,u2,…,u n ),u n Represents a word or subword so that it can be input into the model for processing; data cleaning and denoising improves the quality of data by cleaning and removing some useless noise data, such as HTML tags, special characters, etc. The above operations will help improve the performance and accuracy of the model, making the text prediction results more accurate and practical.
[0037] S2. The large language model is based on the Transformer architecture and gives a set of target vocabulary T (t1, t2, ... t n ) and probability P(p1,p2,…p n ),
[0038] Transformer is a neural network architecture for natural language processing proposed by Google in 2017. It uses the self-attention mechanism to process the dependencies between words in the input sequence, allowing the model to take into account the contextual information of the entire input sequence at the same time.
[0039] In step S2, the large language model selects a large language model encoding algorithm based on the Transformer architecture for sample training; based on a natural language vectorization algorithm, the keywords are analyzed and converted into corresponding word embedding vectors;
[0040] The Transformer architecture includes an encoder and a decoder; the encoder and the decoder include an attention layer and a feedforward neural network layer; an error correction filter is added after the Self-Attention layer.
[0041] The decoding process of the large language model includes the following steps:
[0042] The decoder first applies a self-attention mechanism to the currently generated output sequence; calculates the dependency between the previous and next words;
[0043] The decoder interacts the state of the current decoding position with all hidden states generated by the encoder, and obtains relevant context information of the text sentence based on the encoder-decoder attention layer;
[0044] The decoder passes the combined vector to the feed-forward neural network layer;
[0045] The attention layer and the feedforward neural network layer add residual connections to retain the original input information; stabilize the training process based on layer normalization and optimize the gradient propagation of the text;
[0046] The decoder will go through a linear layer and generate a probability distribution vector based on the loss function, which generates the next target word and its probability.
[0047] In order to ensure the accuracy and fluency of the output results, the target vocabulary set needs to be filtered once. The error correction filter usually checks whether the replaced text conforms to the grammatical and semantic rules, and performs further error correction and correction as needed. For example, if there are repeated words, ungrammatical sentence structures, or obvious semantic errors in the replaced text, the error correction result filter will correct these problems. After being screened and corrected by the error correction filter, the final output text should be a fully corrected text that conforms to the grammatical and semantic rules and can more accurately express the original meaning.
[0048] S3. According to Chinese expression habits, formulate the rules of word order expression order, and introduce dynamic adjustment factors according to the rules to adjust the probability of target vocabulary; repeat steps S2-S3 until the current target vocabulary meets the requirements;
[0049] The order of morphemes and word combinations in a language. Including the order of subject, predicate and object, the order of modifiers and central words, and the order of modifiers;
[0050] According to the order of subject, predicate and object, there are six types: SOV, SVO, VSO, VOS, OSV and OVS. Word order is a major means of expression in Chinese grammatical structure. The same words have different order, different syntactic structure and different meanings.
[0051] S31. Determine the current target vocabulary based on the existing text information and expression habits i The target word with the largest probability is selected. If the target word's part of speech matches the expression habit, no adjustment is required and the target word is used directly. If the target word's part of speech does not match the expression habit, the next target word is selected and step S31 is repeated until the target word's part of speech matches the expression habit, and its probability is adjusted using a dynamic adjustment factor, the formula is as follows:
[0052] p′ i =p i +(p max -ln2 r )>p max
[0053] In the formula, p i is the corresponding probability in the target vocabulary, r is a dynamic adjustment factor (0, 1); p max is the maximum value in the probability set.
[0054] In step S31, the large language model sets the probability of the target words that were not selected in the previous round to zero, determines that the current target word is unavailable, and performs pruning.
[0055] S32. Change p′ i Feedback is given to the large language model to re-output a new round of target vocabulary sets and their probabilities.
[0056] S4. Output the result according to the preset template.
[0057] Embodiment 1:
[0058] I like watching ____, especially Calabash Brothers.
[0059] Target vocabulary set: (movies, works, food, yoga...),
[0060] Probability: (0.9, 0.08, 0.01, ...),
[0061] Based on expression habits and the principle of maximum probability, choose "movie".
[0062] Embodiment 2:
[0063] I am doing ____ homework, which involves many functional relationships.
[0064] Target vocabulary set: (Chinese, mathematics, aerospace...),
[0065] Probability: (0.8, 0.18, 0.01, ...),
[0066] Based on the expression habits and context information, select "Mathematics".
[0067] At this time, the probability of language is set to zero, and the probability of mathematics is calculated using p′ i Recalculate: r = 0.2
[0068] p′ i =0.18+(0.8-0.13)=0.84
[0069] 0.84 is input into the Transformer, the system outputs “math”, and the basis probabilities for the next set of target vocabulary sets are recalculated.
[0070] It is very important to evaluate the performance and optimization effect of the new model. In the evaluation, we need to use a set of predefined test sets to measure the performance of the model. The test set should contain a series of text samples that contain known errors and have been manually annotated with the correct correction results. During the evaluation process, we will input each text sample in the test set into the model and calculate the accuracy, recall, F1 value and other indicators between the model's prediction results and the manually annotated results. Accuracy refers to the proportion of correct results predicted by the model to all predicted results; recall refers to the proportion of errors that the model can correctly identify to all known errors; F1 value is the harmonic mean of accuracy and recall. The evaluation results will help us identify and solve problems in the model and optimize the performance of the model to achieve higher text error correction accuracy, as shown in Table 1.
[0071] Error correction method Precision Recall F1 T5 0.53 0.48 0.52 Macbert 0.81 0.71 0.75 This patent 0.96 0.90 0.92
[0072] A device for generating image text based on deep learning, comprising:
[0073] The acquisition module is used to obtain the word embedding vector of the keyword;
[0074] Extraction module, used to extract features through transformer, target vocabulary T(t1,t2,…t n ) and probability P(p1,p2,…p n );
[0075] Judgment module, judge the current target vocabulary t i Does it conform to current expression habits?
[0076] Generate module and output results according to preset template.
[0077] An electronic device comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the Chinese text generation method as described in the present application.
[0078] A computer-readable medium stores a computer program, wherein the computer program, when executed by a processor, implements the Chinese text generation method of the present application.
[0079] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A text generation method based on a large language model: characterized in that, The following steps are involved: S1. Preprocess the input keywords and convert them into corresponding word embedding vectors; S2. The large language model is based on the Transformer architecture and gives a set of target vocabulary T (t1, t2, ... t n ) and probability P(p1,p2,…p n ), and add an error correction filter after the self-attention layer of the Transformer architecture; set a dynamic adjustment factor in the error correction filter; S3 dynamically adjusts the target vocabulary probability; repeats steps S2-S3 until the current target vocabulary meets the requirements; S4. Output the result according to the preset template.
2. A text generation method based on a large language model according to claim 1: characterized in that, In step S2, the large language model selects a large language model encoding algorithm based on the Transformer architecture for sample training; Based on the natural language vectorization algorithm, the keywords are analyzed and converted into corresponding word embedding vectors; The Transformer architecture includes an encoder and a decoder; the encoder and the decoder include an attention layer and a feedforward neural network layer; the decoding process of the large language model includes the following steps: The decoder first applies a self-attention mechanism to the currently generated output sequence; calculates the dependency between the previous and next words; The decoder interacts the state of the current decoding position with all hidden states generated by the encoder, and obtains relevant context information of the text sentence based on the encoder-decoder attention layer; The decoder passes the combined vector to the feed-forward neural network layer; The attention layer and the feedforward neural network layer add residual connections to retain the original input information; Stabilizing the training process and optimizing the gradient propagation of the text based on layer normalization; The decoder will go through a linear layer and generate a probability distribution vector based on the loss function, which generates the next target word and its probability.
3. A text generation method based on a large language model according to claim 1: characterized in that, In step S3, S31. Determine the current target vocabulary based on the existing text information and expression habits i Part of speech; Select the target word with the highest probability. If the part of speech of the target word matches the expression habit, no adjustment is required and the target word is used directly. If the part of speech of the target word does not match the expression habit, select the next target word and repeat step S31 until the part of speech of the target word matches the expression habit, and adjust its probability using the dynamic adjustment factor. The formula is as follows: p′ i =p i +(p max -ln2 r ; In the formula, p i is the corresponding probability in the target vocabulary, and r is a dynamic adjustment factor. S32. Change p′ i Feedback is given to the large language model to re-output a new round of target vocabulary sets and their probabilities.
4. A text generation method based on a large language model according to claim 3: characterized in that, In step S31, the large language model sets the probability of the target words that were not selected in the previous round to zero, determines that the current target word is unavailable, and performs pruning.
5. A text generation method based on a large language model according to claim 3: characterized in that, In step S2, the error correction filter will check whether the replaced text conforms to grammatical and semantic rules, and perform further error correction and correction as needed: if the replaced text contains repeated words, ungrammatical sentence structures or obvious semantic errors, the error correction result filter will correct these problems.
6. A device for a text generation method based on a large language model according to any one of claims 1 to 5, characterized in that: include: The acquisition module is used to obtain the word embedding vector of the keyword; Extraction module, used to extract features through transformer, target vocabulary T(t1,t2,…t n ) and probability P(p1,p2,…p n ); Judgment module, judge the current target vocabulary t i Does it conform to current expression habits? Generate module and output results according to preset template.
7. An electronic device comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the text generation method according to any one of claims 1 to 5.
8. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the text generation method according to any one of claims 1 to 5 is implemented.