Intelligent text multi-label method for driving GPT small model by large language model
By designing the propt process and combining the semantic retrieval model, using the large language model to pre-label the small GPT model, the problems of limited inference speed of large language model and high cost of development of GPT small model are solved, and high-quality multi-label generation and rapid analysis are achieved. Suitable for government affairs applications with a large number of netizens’ messages.
Patent Information
- Application Number
- CN202410774674.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-17
- Publication Date
- 2024-11-15
AI Technical Summary
Large language model has limited reasoning speed in the analysis of massive people's livelihood demands and netizens' message analysis scenarios. Due to the high development cost and poor understanding ability, it is difficult to apply it in the government field, and the existing technology is difficult to effectively generate high-quality multi-dimensional models. Label.
By designing a suitable propt process combined with semantic retrieval model, using large language models to pre-notify the small GPT model, build an intelligent text multi-label dataset, and adopting KV Cache and FlashAttention technology to reduce dimensionality and punishment factors to improve the performance and inference speed of the small GPT model .
It realizes high-quality multi-label generation, reduces the difficulty of recommending tags in large language models, improves the quality of pre-labeled message content and the reliability of generating tags, simplifies code length, improves reasoning speed, and is suitable for the massive netizens' messages. Quick analysis.
Abstract
Description
Technical Field
[0001] The present invention relates to the field of AI language model driven technology, and specifically to an intelligent text multi-label method in which a large language model drives a GPT small model. Background Art
[0002] With the maturity of large language model technology, artificial intelligence has entered the era of general artificial intelligence and artificial intelligence generation. The general text analysis and text reasoning capabilities of large language models have greatly improved the promotion and application of artificial intelligence in various fields. However, large language models such as Llama 2 and ChatGLM3-6B have very good effects, but the model parameters of large language models occupy a large storage space. At the same time, when applied to scenarios with high concurrency requirements and fast speed requirements, their speed is gradually approaching the limit with the introduction of acceleration technologies such as KVCache and FlashAttention. For scenarios that require fast analysis, such as analysis of massive livelihood appeals and analysis of netizens' messages, the promotion and application of large language models is greatly limited; for the GPT small model, although the model parameters are small and the reasoning speed is fast, because it requires a large amount of labeled data, the development cost is high, and the model parameters are small and the understanding ability is poor, so the application of the GPT small model in the government affairs field is extremely difficult. This patent analyzes the advantages and disadvantages of the large language model and the small GPT model, and finally uses the small GPT model to provide a technical solution for providing services. This patent mainly uses this method to solve the problem of intelligently recommending multiple tags for netizens' messages. Intelligent multi-tags are to add multiple tags to the message content of a netizen. When all netizens' messages are marked with corresponding tags, the message content of each tag can be counted for in-depth analysis, so it has a wide range of application value. At the same time, the current industry language model-related technologies have undergone earth-shaking changes. This patent has made relevant technical improvements to the main modules of the small GPT language model to improve the performance of the small GPT language model. Summary of the invention
[0003] This patent proposes a method for pre-labeling a small GPT language model intelligent text multi-label dataset based on a large language model, that is, by designing a suitable prompt combined with a semantic retrieval model, multiple labels are generated for the content of netizens' messages in the database. The innovation of this patent lies mainly in the design of a novel prompt process to construct a dataset that generates multiple labels from netizens' message content. The specific implementation method of the GPT small language model that is compatible with KV Cache, FlashAttention, dimensionality reduction, penalty factor and other technologies continuously improves the performance of the small GPT model.
[0004] An intelligent text multi-label method in which a large language model drives a GPT small model includes the following steps: S1. Text data source It includes message text and text tags. Text tags are given according to the semantics of the message text. The text tags form a text tag library. S2. Training set construction The large language model is used to establish a corresponding relationship between the message text and the text tag library to form a small model GPT training data set. Based on the message content, the corresponding text tag is recommended through the semantic retrieval small model text2vec-base-chinese-paraphrase. Because the message content is a long text with a length of 30 to 50 words, the recommended corresponding text tag is a short text with a length of less than 10 words. The short text uses the open source large language model Qwen1.5-14B or ChatGLM3-6B to interpret all text tags through prompt event tags to form a long text, forming a mapping relationship between the message text and the text tag interpretation; after the tag interpretation, through the semantic retrieval model, text2vec-base-chinese-paraphrase recommends multiple tags, and uses prompt to explain the event tags and message content. First, all the explanations of the event tags are vectorized through text2vec-base-chinese-paraphrase, and then the message text is also vectorized through text2vec-base-chinese-paraphrase. Then, semantic vectorization is used to retrieve multiple most relevant event tag explanations, and multiple corresponding tags are taken out. All message texts are processed using the above method to form a corresponding training set, and the formed training set is used to train the small GPT model; S3. Create a token Training the GPT model based on message text requires making corresponding tokens and collecting tokens of the open source large language model, where n_tokens is the number of tokens in the open source large model, that is, collecting as many tokens of the open source large model as possible, and then putting them into a file token_LLM.txt.txt, then tokenizing all the message texts, counting the high-frequency words of the tokens, retaining 40,000 tokens, and counting the high-frequency tags of the netizen message event tags, retaining 8,000 tags as tokens, and putting them in a file as a vocabulary. At the same time, [CLS] is used as the text start token in GPT model training, [PAD] is the text filler for GPT model texts that are not long enough, [UNK] is the symbol that does not appear in the definition dictionary of the netizen message content, and [SEP] is the text separator; S4. Building the GPT model The message text {Text} and text label {Labels} data are segmented by [SEP] and used as training data for GPT; S5. Prediction of appeal content labels The GPT model is used to generate multiple text labels from the message text. All labels are tokenized through the produced tokens and then put into the label token list. In the process of generating multiple labels with the small model GPT, the probability of each GPT token candidate that is not in the label token list is penalized by a coefficient, that is, multiplied by a coefficient of 0.9, and the generated token does not belong to the label token list.
[0005] In S2, multiple tags are recommended, and the number of the multiple tags is 20.
[0006] The training data set in S4 is processed by the GPT model, and the implementation method of the causal attention module sub-layer is described as follows: the semantic representation is obtained by reducing the dimension to the d_small dimension to obtain the key tensor, the semantic representation is obtained by reducing the dimension to the d_small dimension to obtain the query tensor, and the semantic representation is obtained by maintaining the dimension to the d_model dimension to obtain the value tensor. The key and value are determined according to the training stage and the inference stage to determine whether they need to be cached.
[0007] The beneficial effects of the present invention are: A method for constructing a large language model to generate a high-quality multi-label dataset is proposed. Although each technical point of the solution proposed in this patent is well known to everyone, this patent can realize the construction method of a training set by pre-labeling netizens' messages and label library through a large model through careful analysis and systematic prompt design, which solves the uncontrollability of semantic retrieval recommendation labels. The technical problem of searching for short labels in long message content in semantic retrieval is avoided to a certain extent through prompt instruction interpretation. A certain number of labels are pre-recommended through semantic retrieval, which reduces the difficulty of recommending labels by a large language model and improves the quality of labels generated from pre-labeled message content.
[0008] This patent migrates the existing large model technology to the GPT small language model technology through multiple practices, achieving concise code length. Because it adopts KV Cache and Flashattention commonly used in large language models, the inference speed is fast.
[0009] This patent implements a specific method for making small language model tokens based on the tokens of an open source large model for a small model in a dedicated domain. It has stronger problem-specificity and makes tokens based on actual corpus. At the same time, it aims at the stability of GPT generation. If the probability of the generated token is not in the label dictionary, a token penalty will be performed to improve the reliability and stability of the generated labels. DETAILED DESCRIPTION
[0010] An intelligent text multi-label method in which a large language model drives a GPT small model includes the following steps: S1. Text data source It includes message text and text tags. Text tags are given according to the semantics of the message text. The text tags form a text tag library. S2. Training set construction Use the large language model to establish a correspondence between the message text and the text tag library to form a small model GPT training data set. Based on the message content, the corresponding text tag is recommended through the semantic retrieval small model text2vec-base-chinese-paraphrase. Because the message content is a long text with a length of 30 to 50 words, the recommended corresponding text tag is a short text with a length of less than 10 words. The short text uses the open source large language model Qwen1.5-14B or ChatGLM3-6B to interpret all text tags through prompts for event tags to form a long text, forming a mapping relationship between the message text and the text tag interpretation; After the label interpretation and semantic retrieval model, text2vec-base-chinese-paraphrase recommends multiple labels, and the event label and message content are interpreted by prompt. First, the event label interpretation is vectorized by text2vec-base-chinese-paraphrase, and then the message text is also vectorized by text2vec-base-chinese-paraphrase. Then, semantic vectorization is used to retrieve multiple most relevant event label interpretations, and multiple corresponding labels are taken out. All message texts are processed by the above method to form a corresponding training set, and the formed training set is used to train the small GPT model; S3. Create a token Training the GPT model based on message text requires making corresponding tokens and collecting tokens of the open source large language model, where n_tokens is the number of tokens in the open source large model, that is, collecting as many tokens of the open source large model as possible, and then putting them into a file token_LLM.txt.txt, then tokenizing all the message texts, counting the high-frequency words of the tokens, retaining 40,000 tokens, and counting the high-frequency tags of the netizen message event tags, retaining 8,000 tags as tokens, and putting them in a file as a vocabulary. At the same time, [CLS] is used as the text start token in GPT model training, [PAD] is the text filler for GPT model texts that are not long enough, [UNK] is the symbol that does not appear in the definition dictionary of the netizen message content, and [SEP] is the text separator; S4. Building the GPT model The message text {Text} and text label {Labels} data are segmented by [SEP] and used as training data for GPT; S5. Prediction of appeal content labels The GPT model is used to generate multiple text labels from the message text. All labels are tokenized through the produced tokens and then put into the label token list. In the process of generating multiple labels with the small model GPT, the probability of each GPT token candidate that is not in the label token list is penalized by a coefficient, that is, multiplied by a coefficient of 0.9, and the generated token does not belong to the label token list.
[0011] In S2, multiple tags are recommended, and the number of the multiple tags is 20.
[0012] The training data set in S4 is processed by the GPT model, and the implementation method of the causal attention module sub-layer is described as follows: the semantic representation is obtained by reducing the dimension to the d_small dimension to obtain the key tensor, the semantic representation is obtained by reducing the dimension to the d_small dimension to obtain the query tensor, and the semantic representation is obtained by maintaining the dimension to the d_model dimension to obtain the value tensor. The key and value are determined according to the training stage and the inference stage to determine whether they need to be cached.
[0013] The following is described in conjunction with specific embodiments: The Chinese text data sources of S1 are: 1. Netizen messages in the netizen message system in government affairs processing, usually ranging from dozens of words to 700 words, usually millions of netizen messages, for example, a message data is "At present, XXX and 3 other persons suspected of violating the law are respectively detained for 15 days and fined 1,800 yuan, and XXX and 3 other persons are respectively detained for 5 days", where XXX is the name hidden for data security; 2. The event label library established by experts in the field of government affairs message processing, usually hundreds of thousands, for example, labels include criminal detention, administrative detention, administrative punishment, etc., which are usually constructed manually.
[0014] The data elements in S2. cannot be used directly to develop an intelligent multi-label language model. It is necessary to establish a relationship between the data elements and the corresponding labels in order to train the small language model GPT. This patent mainly uses a large language model to establish a relationship between the two data elements of netizens' message content and the label library to form the final small model GPT training data set.
[0015] Specifically, the event label of the message is concise and to the point. Based on the message content, the semantic retrieval model text2vec-base-chinese-paraphrase can recommend the corresponding event label. Because the message content is a long text - between 30 and 50 words, and the event label is a short text - a label within 10 words, there will be a problem of long text searching for short text. Because the matching corpus of long and short texts is relatively lacking, the semantic retrieval model is not effective. At the same time, event labels are artificially accumulated and summarized over a long period of time, so they are more written, while the message content is seriously spoken, that is, there is a semantic gap. Therefore, it is necessary to use an open source large language model similar to Qwen1.5-14B, ChatGLM3-6B, etc. to interpret all event labels through prompts to form long texts. If If an event label is {Text}, then add the prompt instruction {\nPlease explain the above label in detail, with 30-50 words}, and then concatenate {Text} and prompt together to form a mapping relationship between the event label and the corresponding event label explanation. For example, after the above prompt template is input into the large model, the explanation of "export tax rebate" is output as "Export tax rebate refers to the policy of the Chinese government to return part of the value-added tax, consumption tax and other taxes paid on exported goods to enterprises in order to encourage exports and increase the international competitiveness of export enterprises. This policy effectively reduces the operating costs of enterprises, speeds up the turnover of funds, and provides strong support for the development of my country's foreign trade."; "Export tax rebate" is a label, and there is a one-to-one mapping relationship between the explanations of "export tax rebate".
[0016] 2. Text multi-label. After the above label explanation, it is completely possible to recommend multiple labels through semantic retrieval models, such as text2vec-base-chinese-paraphrase. For example, the current message recommends similar label explanations through the semantic retrieval model, and then the label corresponding to the label explanation is taken out, that is, the text multi-label task is completed. However, there is a natural semantic gap between the label explanation and the content of netizens' messages, that is, the content of netizens' messages is usually too colloquial, and the explanation of event labels is usually more written, and the recommendation effect is extremely poor. Entering the era of large models, the powerful semantic understanding and decision-making ability of large language models is well known to the public. If the event label and the message content are passed through prompt, let the large model directly recommend labels based on semantics, usually within 20, and the specific number can be increased or decreased according to the server performance, and then let the large model use its own advantages to select labels from 20 labels, it will further improve the accuracy of text recommendation labels. If all labels, usually hundreds of thousands, are directly written into prompt, it will exceed the acceptable context length of the large language model. In order to solve this problem, this patent adopts a semantic retrieval model to first roughly recommend multiple labels, such as 20 labels, and then adopts a large model to select labels. The specific implementation is as follows: First, all the explanations of event tags are vectorized through text2vec-base-chinese-paraphrase. Then, a piece of content from a netizen is also vectorized through text2vec-base-chinese-paraphrase. Then, semantic vectorization is used to retrieve the 20 most relevant explanations of event tags, and the corresponding tags of the tag explanations are taken out. All the netizen comments are processed in the same way, so that 20 tags are recommended for all the netizen comments. Usually, a relationship is established between the netizen comments and some tags, but it is not completely accurate. Using the powerful understanding ability of the large language model, carefully select labels from 20 labels through the prompt command. If a netizen's message is {Text}, the corresponding 20 labels are {Labels}, and spaces are kept between the 20 labels, then the prompt command is designed: You are an expert who has worked in the government field for many years and has rich experience in solving netizen message problems. Please label the netizen's message content according to the following netizen message content. Keep spaces between labels and do not have redundant content. The message content is: \n{Text}\nThe reference label provided is: \n{Labels}\n. Through the above ingenious design, the netizen's message content can be labeled with corresponding labels, thereby forming a corresponding training set. However, the netizen's message content is usually about 50,000 per day. Therefore, if the large model is directly deployed in the above way, it will waste a lot of computing resources, the cost is high, and it cannot be analyzed quickly. In order to solve this problem, this patent uses a small language model GPT to train a small GPT model based on the training set formed above.
[0017] Creating tokens in S3 is a common technology in the industry. In order to ensure the integrity of this patent, it is specially explained that training the GPT model based on the content of netizens' messages requires creating corresponding tokens and widely collecting tokens of open source large language models. Taking the Gemma model as an example, the specific operations are as follows: from gemma.tokenizer import Tokenizer tokenizer = Tokenizer('tokenizer.model') n_tokens = 256000 gen = [] for aa in range(n_tokens): gen.append(tokenizer.decode([aa])+'\n') fo1 = open('token_LLM.txt', "w", encoding='utf-8') fo1.writelines(gen) Among them, n_tokens is the number of tokens in the open source large model, that is, collect as many tokens of the open source large model as possible, and then put them into a file token_LLM.txt.txt, then all the content of netizens' messages are tokenized, and the high-frequency words of the tokens are counted, and about 40,000 tokens are retained. At the same time, the high-frequency tags of the event tags of netizens' messages are counted, and about 8K tags are retained as tokens. About 40,000 tokens and about 8K tokens are placed in a file as a vocabulary. At the same time, [CLS] is used as the text start token in GPT model training, [PAD] is the text filler for GPT model length that is not enough, [UNK] is the netizen message content that does not appear in the definition dictionary, and [SEP] is the text separator. Through the above operations, token characters for the content of netizens' messages are formed, reducing the training cost. Some event tags are also used as tokens, thereby reducing the fragmentation of event tags in the generation process of the GPT small language model. Here is the implementation method of making small model tokens proposed by this patent. Targeted token production avoids the waste of resources by irrelevant tokens.
[0018] S4. The data of message text {Text} and text label {Labels} are segmented through [SEP] as the training data of GPT. How to import the training data set into the GPT model is the most mature technical point in the industry. How to train the GPT model is the continuation of this patent technology. The main protection point of this patent is to realize a very simple small model integration large model technology implementation method. The structure of each sub-layer of the GPT overall model is as follows: dec_inputs = RMSNorm1(dec_inputs) dec_inputs = residual+dec_self_attn(dec_inputs,clean_cache) residual = dec_inputs dec_inputs = RMSNorm2(dec_inputs) dec_outputs = residual+feed_forward(dec_inputs) dec_inputs is the input of the GPT model sublayer, RMSNorm is variance normalization, dec_self_attn is causal attention, and feed_forward is a feedforward neural network. This patent mainly implements the technology for the attention part of the GPT network. This patent mainly proposes a scaled_dot_product_attention attention implementation method based on torch-based flash attention. Only a few lines of code can integrate dimensionality reduction technology, KV Cache cache acceleration technology, and flash attention technology. Compared with the current mainstream solutions for large models in the industry (KV Cache, flashattention, multi query attention), the code length is short and the readability is high. The language description of the causal attention module sublayer implementation method is as follows: The semantic representation is obtained by reducing the dimension to d_small to obtain the key tensor The semantic representation is obtained by reducing the dimension to d_small to obtain the query tensor The semantic representation is obtained by keeping the dimension to the d_model dimension to obtain the value tensor key, value determines whether it needs to be cached according to the training phase and the inference phase The scaled_dot_product_attention (i.e. flash attention) in torch.nn.functional is used to calculate the new semantic representation of the protection attention coefficient.
[0019] The GPT model structure of this patent is different from the previous GPT model in that the multi-head mechanism is deleted. The multi-head operation in the multi-head mechanism wastes computing time, so this patent deletes the multi-head mechanism. The implementation method of this patent is more readable than the existing open source large model code at the code level, and the implementation is more concise, and it integrates the key technologies of the large model KV Cache and flash attention. The industry's multi-query shared parameters and multi-attention parameters have many operation steps, which belong to the category of dimensionality reduction. The code implementation is complex and the readability is extremely low. The implementation method of this patent greatly reduces the migration ability of large model technology to small model technology. The other parts of GPT are not the key points of this patent, and this patent will not elaborate on them.
[0020] GPT in S5 is a text generation model. The GPT model trained by the above method can fully realize the generation of multi-labels from netizens' messages with high accuracy. However, in order to further improve the stability of intelligent generation of multi-labels, that is, to ensure that the generated content must be the label content accumulated by humans over a long period of time, all the labels of netizens' messages are tokenized through the production of tokens. In the process of generating multi-labels with the small model GPT, the probability of each GPT token candidate that is not in the label token list is penalized by a coefficient, that is, multiplied by a coefficient of 0.9. That is, if the generated token does not belong to the label token list, the possibility of GPT predicting the token is reduced, thereby ensuring that GPT generates event labels in the message event library as much as possible.
[0021] Glossary Large language model: A large language model is a deep learning model that can be pre-trained with a large amount of text data to generate natural language text or understand the meaning of language text. This model can handle a variety of natural language tasks, such as text classification, question answering, dialogue, text multi-labeling, etc., and is an important path to general artificial intelligence; GPT: GPT is a text generation model based on the Transformer decoder. The GPT model can predict the next word or sentence based on the context and can be used in applications such as automatic writing, machine translation, and dialogue systems. Causal attention mechanism: The causal attention mechanism is an attention mechanism in natural language processing (NLP). It follows the causal relationship when processing text token sequence data, that is, it assumes that the tokens in the token sequence have an impact in chronological order, and the current token generated will not be affected by the subsequent tokens.
[0022] Intelligent text multi-labeling: In the field of artificial intelligence, multi-label text classification is a common task, which means labeling a text into multiple categories. Text multi-labeling is widely used in many fields such as government big data analysis, public opinion analysis of people's livelihood events, and information retrieval.
Claims
1. An intelligent text multi-label method using a large language model to drive a small GPT model, characterized in that The steps include: S1. Text data source It includes message text and text tags. Text tags are given according to the semantics of the message text. The text tags form a text tag library. S2. Training set construction Use the large language model to establish a correspondence between the message text and the text tag library to form a small model GPT training data set. Based on the message content, the corresponding text tag is recommended through the semantic retrieval small model text2vec-base-chinese-paraphrase. Because the message content is a long text with a length of 30 to 50 words, the recommended corresponding text tag is a short text with a length of less than 10 words. The short text uses the open source large language model Qwen1.5-14B or ChatGLM3-6B to interpret all text tags through prompts for event tags to form a long text, forming a mapping relationship between the message text and the text tag interpretation; After the label interpretation and semantic retrieval model, text2vec-base-chinese-paraphrase recommends multiple labels, and the event label and message content are interpreted by prompt. First, the event label interpretation is vectorized by text2vec-base-chinese-paraphrase, and then the message text is also vectorized by text2vec-base-chinese-paraphrase. Then, semantic vectorization is used to retrieve multiple most relevant event label interpretations, and multiple corresponding labels are taken out. All message texts are processed by the above method to form a corresponding training set, and the formed training set is used to train the small GPT model; S3. Create a token Training the GPT model based on message text requires making corresponding tokens and collecting tokens of the open source large language model, where n_tokens is the number of tokens in the open source large model, that is, collecting as many tokens of the open source large model as possible, and then putting them into a file token_LLM.txt.txt, then tokenizing all the message texts, counting the high-frequency words of the tokens, retaining 40,000 tokens, and counting the high-frequency tags of the netizen message event tags, retaining 8,000 tags as tokens, and putting them in a file as a vocabulary. At the same time, [CLS] is used as the text start token in GPT model training, [PAD] is the text filler for GPT model texts that are not long enough, [UNK] is the symbol that does not appear in the definition dictionary of the netizen message content, and [SEP] is the text separator; S4. Building the GPT model The message text {Text} and text label {Labels} data are segmented by [SEP] and used as training data for GPT; S5. Prediction of appeal content labels The GPT model is used to generate multiple text labels from the message text. All labels are tokenized through the produced tokens and then put into the label token list. In the process of generating multiple labels with the small model GPT, the probability of each GPT token candidate that is not in the label token list is penalized by a coefficient, that is, multiplied by a coefficient of 0.9, and the generated token does not belong to the label token list.
2. According to claim 1, a large language model drives the GPT small model intelligent text multi-label method, characterized in that In S2, multiple tags are recommended, and the number of the multiple tags is 20.
3. According to claim 1, a large language model drives the GPT small model intelligent text multi-label method, characterized in that The training data set in S4 is processed by the GPT model, and the implementation method of the causal attention module sub-layer is described as follows: the semantic representation is obtained by reducing the dimension to the d_small dimension to obtain the key tensor, the semantic representation is obtained by reducing the dimension to the d_small dimension to obtain the query tensor, and the semantic representation is obtained by maintaining the dimension to the d_model dimension to obtain the value tensor.
Citation Information
Cited By
Text multi-label classification method and device, electronic equipment and storage medium
CN119807417A
