Keyword extraction method based on understanding ability of large language model

Abstract

The invention relates to the technical field of large language model processing, in particular to a keyword extraction method based on the understanding ability of a large language model. According to the patent, the TF-IDF value of the vocabulary is taken as a starting point, the TF-IDF value is continuously adjusted by means of a large language model, at the moment, the TF-IDF value does not belong to an original TF-IDF definition, but is a value reflecting the importance of the vocabulary after being optimized by the large language model, and the patent is recorded as an importance weight; the method mainly aims at forming a method for continuously correcting the importance of vocabularies by means of a large language model, continuously expanding the vocabularies, forming a keyword extraction method driven by a large model technology, fully utilizing the semantic comprehension ability of the large language model, and fully utilizing the characteristic of high speed of traditional keyword extraction. The user finally deploys the keyword extraction capability based on the dictionary word weight, the keyword extraction capability is provided for the user to use, the speed is far higher than the keyword extraction capability of a large language model by means of a prompt instruction, and the effect is stable.

Description

Technical Field

[0001] The invention relates to the technical field of large language model processing, and in particular to a keyword extraction method based on the understanding ability of a large language model. Background Art

[0002] The background of keyword extraction technology can be traced back to the development of information retrieval, text mining and natural language processing. In the early information retrieval system, how to accurately extract keywords is crucial to the system when retrieving documents through keywords. Especially now that RAG technology based on large models is widely used, the importance of keyword extraction is becoming more and more prominent. The development of keyword extraction technology provides a very effective means for information retrieval, RAG retrieval and large model question and answer. Keyword retrieval and semantic retrieval are an important part of RAG technology. With the rapid growth of the Internet and digital information, people need to extract useful information from massive text data for analysis, mining and decision-making. With the popularization and use of large model RAG technology, users need to be able to quickly understand the theme and content of search results, and keyword extraction technology can help search engines extract the main keywords of documents and improve the accuracy of searches. At the same time, with the continuous advancement of natural language processing technology, especially large language model-related technologies, the ability to understand text semantics has been improved, providing more optimization possibilities for the development of keyword extraction technology. Summary of the invention

[0003] Keyword extraction plays an important role in information retrieval, information mining, information decision-making, and RAG knowledge question-answering. Therefore, generations of natural language practitioners have spent a lot of effort to establish the TF-IDF value of keywords. These TF-IDF values ​​represent the importance of a word. Entering the era of large models, especially large language models, have had a significant impact on keyword extraction technology. Many artificial intelligence practitioners have begun to use large language models combined with prompt engineering to extract keywords, because large language models learn the semantics and context of language in massive corpora, so that they can generate keywords with semantic understanding, and the keyword extraction effect is more obvious. However, keyword extraction is usually an intermediate technical link in the product system. If a large language model is directly deployed to extract keywords, the mathematical computing ability, logical reasoning ability, and analytical decision-making ability that the large language model should have demonstrated through its emergence ability will be occupied by computing resources, affecting the user's instant interactive experience. At the same time, the output format of ordinary large models with smaller parameter scales is usually not stable enough. This patent aims to make full use of the large language model technology to optimize and improve the traditional TF-IDF-based keyword extraction technology. It takes the TF-IDF value of the vocabulary as the starting point and continuously adjusts the TF-IDF value with the help of the large language model. At this time, the TF-IDF value no longer belongs to the original TF-IDF definition, but is a value that reflects the importance of the vocabulary after optimization by the large language model. This patent is recorded as the importance weight. This patent mainly uses the large language model to form a method for continuously correcting the importance of vocabulary, while continuously expanding the vocabulary to form a keyword extraction method driven by large model technology, making full use of the semantic understanding ability of the large language model, and at the same time making full use of the fast speed of traditional keyword extraction. Users finally deploy keyword extraction capabilities based on dictionary word weights and provide them to users. The speed is much faster than the keyword extraction capability of the large language model with the help of prompt instructions, and the effect is stable.

[0004] The following technical solutions are used to solve the above problems: A keyword extraction method based on the understanding ability of a large language model includes the following steps: S1. Create a large vocabulary Collect relevant words from natural language processing word segmentation tools such as Jieba, summarize them into the list words, and remove duplicate words; S2. Collect and summarize TF-IDF text corpus TF-IDF is produced using a large language model pre-training dataset, and TeleChat-PTD is produced using the TeleChat pre-training dataset; S3. Create the TF-IDF value for each word in S1 The TF value is taken as the average value method, and the words in any specific text in the large language model pre-training dataset are recorded as , the normalized word frequency is expressed as: In the above formula yes In a training corpus The number of occurrences in the denominator In this training corpus The sum of the length of characters; through a text will get a file a word , count the number of times the word appears in all pre-training corpus texts , take the average value of all pre-training corpus texts as the TF value of the word, and get the TF value of all words through the above method. The TF value is written as uppercase ; Inverse Document Frequency (IDF) is a measure of the general importance of a word. The total number of texts in the training set is divided by the number of training texts containing the word, and the logarithm of the quotient is obtained: : The total number of training texts in the pre-training corpus : Contains words The number of files is to prevent the divisor from being zero, so it is generally used ; The TF-IDF value of each word can be obtained through the above method; S4. Create vocabulary importance_score value The TF-IDF value of each word represents the initial value of the word's importance weight, and the keyword extraction capability can be completed based on the TF-IDF value of each word. However, the words collected in step S1 are corrected in their TF-IDF values ​​and new words are expanded.

[0005] The S4 specifically includes the following steps: step1 The TF-IDF value range is modified. The maximum value of all words in S1 is recorded as max, and the minimum value is recorded as min. Then it is linearly mapped to 5-100 points. The mapping method is as follows: Thus, all TF-IDF values ​​are mapped to 5 to 100 points; Step 2 Traverse each text in the pre-trained dataset, first extract keywords through the large language model, and then extract keywords based on Tf-idf. For the above large model input, the open source large language model extracts keywords. However, the large language model is usually a generative model, and usually has redundant outputs. Then, through regular matching, obtain the keyword vocabulary based on the large language model, recorded as Keywords_LLM; if the regular matching cannot be parsed, the text will not be processed later; Step 3 Modify the Importance-Score for Keywords_Statistics and Keywords_LLM of each text in the training set; continuously integrate the semantic understanding ability of the big model to complete the expansion of the vocabulary. At the same time, as the number of times the big model extracts keywords, the importance score will gradually increase; Step 4 The maximum importance of all words after adjustment is recorded as Score_max, and the minimum importance is recorded as Score_min, which is linearly mapped to 0-100 points. The mapping method is as follows: Step 5 Map to Importance_Score from 0 to 100 for keyword extraction, and equate Importance_Score to TF-IDF to extract keywords.

[0006] The beneficial effect of this patent is reflected in the proposal of a systematic and implementable technical solution for continuously optimizing the traditional TF-idf keyword extraction based on a large language model. The traditional keyword extraction capability based on TF-IDF has a low vocabulary coverage, and TF-IDF is a statistical indicator. The extracted keywords lack vocabulary understanding ability. In the era of large language models, keywords are extracted through prompt instructions. Although the extracted vocabulary has understanding ability, the format of the extracted keyword results is not stable enough. At the same time, the speed of deployment to customers in the production environment is low. This patent aims to let the large model play a behind-the-scenes role. Through the large language model on the basis of TF-IDF, the implementation method of this patent continuously expands the vocabulary. At the same time, with the help of the semantic understanding ability of the large model, the TF-IDF value is adjusted, so that important words are extracted as keywords by the large language model multiple times, and their importance continues to increase, so that the word importance weight enters the semantic understanding ability of the large model. The implementation plan of this patent is long-term, and in the future it will continue to improve the effect of keyword-based related retrieval capabilities. That is to say, this patent uses a large language model and TF-IDF to coordinate vocabulary, and the Importance_Score obtained will have a long-term effect in the search field and RAG question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 The present invention forms an initial TF-idf word weight schematic diagram; Figure 2 The present invention completes a schematic diagram of updating word weights for pre-training corpus; Figure 3 Schematic diagram of keyword extraction in the actual production environment of the present invention. DETAILED DESCRIPTION

[0008] A keyword extraction method based on the understanding ability of a large language model includes the following steps: S1. Create a large vocabulary Collect relevant words from natural language processing word segmentation tools such as Jieba, summarize them into the list words, and remove duplicate words; S2. Collect and summarize TF-IDF text corpus TF-IDF is produced using a large language model pre-training dataset, and TeleChat-PTD is produced using the TeleChat pre-training dataset; S3. Create the TF-IDF value for each word in S1 The TF value is taken as the average value method, and the words in any specific text in the large language model pre-training dataset are recorded as , the normalized word frequency is expressed as: In the above formula yes In a training corpus The number of occurrences in the denominator In this training corpus The sum of the length of characters; through a text will get a file a word , count the number of times the word appears in all pre-training corpus texts , take the average value of all pre-training corpus texts as the TF value of the word, and get the TF value of all words through the above method. The TF value is written as uppercase ; Inverse Document Frequency (IDF) is a measure of the general importance of a word. The total number of texts in the training set is divided by the number of training texts containing the word, and the logarithm of the quotient is obtained: : The total number of training texts in the pre-training corpus : Contains words The number of files is to prevent the divisor from being zero, so it is generally used ; The TF-IDF value of each word can be obtained through the above method; S4. Create vocabulary importance_score value The TF-IDF value of each word represents the initial value of the word's importance weight, and the keyword extraction capability can be completed based on the TF-IDF value of each word. However, the words collected in step S1 are corrected in their TF-IDF values ​​and new words are expanded.

[0009] The S4 specifically includes the following steps: step1 The TF-IDF value range is modified. The maximum value of all words in S1 is recorded as max, and the minimum value is recorded as min. Then it is linearly mapped to 5-100 points. The mapping method is as follows: Thus, all TF-IDF values ​​are mapped to 5 to 100 points; Step 2 Traverse each text in the pre-trained dataset, first extract keywords through the large language model, and then extract keywords based on Tf-idf. For the above large model input, the open source large language model extracts keywords. However, the large language model is usually a generative model, and usually has redundant outputs. Then, through regular matching, obtain the keyword vocabulary based on the large language model, recorded as Keywords_LLM; if the regular matching cannot be parsed, the text will not be processed later; Step 3 Modify the Importance-Score for Keywords_Statistics and Keywords_LLM of each text in the training set; continuously integrate the semantic understanding ability of the big model to complete the expansion of the vocabulary. At the same time, as the number of times the big model extracts keywords, the importance score will gradually increase; Step 4 The maximum importance of all words after adjustment is recorded as Score_max, and the minimum importance is recorded as Score_min, which is linearly mapped to 0-100 points. The mapping method is as follows: Step 5 Map to Importance_Score from 0 to 100 for keyword extraction, and equate Importance_Score to TF-IDF to extract keywords.

[0010] Example S1: Create a large vocabulary list: In order to ensure the universality of keyword extraction, it is necessary to collect enough words covering the industry field, widely collect relevant words from natural language processing word segmentation tools such as Jieba, and summarize them in a list of words. At the same time, ensure that there is no duplication of words to avoid repeated word frequency statistics.

[0011] S2: Collect and summarize TF-IDF text corpus: Currently, large language models have opened up massive, high-quality, and wide-coverage pre-training data sets. This patent uses large language model pre-training data sets to produce TF-IDF, such as the TeleChat pre-training data set (TeleChat-PTD).

[0012] S3: Create TF-IDF values ​​for each word in S1: The TF value in the TF-IDF value is term frequency (TF), which refers to the frequency of a given word appearing in each training corpus. This patent is to establish a general keyword extraction capability. This patent adopts the method of taking the average value of the TF value. For a certain training corpus: To prevent a certain training corpus from being too long, the TF value needs to be normalized. The same word may have a higher word count in a long training corpus than in a short training corpus, regardless of whether the word is important or not. For any specific word in the large language model pre-training data set For example, its normalized word frequency can be expressed as: In the above formula yes In a training corpus The number of occurrences in the denominator In this training corpus The sum of the length of characters. In order to ensure the statistical speed, this patent does not use the method of counting the number of all words in a training corpus, but normalizes all of them by character length. This does not have much impact on the importance and improves the speed. , count the number of times the word appears in all pre-training corpus texts , take the average value as the TF value of the word. Through the above method, the TF values ​​of all words are obtained. The TF value is written as uppercase .

[0013] Inverse document frequency (IDF) is a measure of the general importance of a word. The total number of texts in the training set is divided by the number of training texts containing the word, and the logarithm of the quotient is obtained: : The total number of training texts in the pre-training corpus : Contains words The number of files is to prevent the divisor from being zero, so it is generally used .

[0014] The TF-IDF value of each word can be obtained through the above method.

[0015] S4: Create the value of Importance_Score of vocabulary importance: The TF-IDF value of each word represents the initial value of the vocabulary importance weight, that is, the keyword extraction capability can be completed based on the TF-IDF value of each word. However, the vocabulary collected in step S1 is usually accumulated by multiple generations of technical personnel over a long period of time, and it is difficult to cover the vocabulary of the new era. At the same time, the TF-IDF value is a statistical indicator. For example, some words themselves appear frequently in a file and in all files, and are important words, but the TF-IDF value of the word is very low, that is, it lacks a certain semantic understanding ability. Therefore, it is necessary to use the semantic understanding ability and the ability to cover new words of the large language model to correct the TF-IDF value and expand new words. The innovation of this patent is mainly to propose a keyword extraction scheme that corrects TF-IDF with the help of a large language model, aiming to correct TF-IDF to a more reasonable importance weight, and with the help of a large language model, give TF-IDF semantic understanding ability, that is, the final vocabulary importance weight, and at the same time improve and expand the dictionary with word importance weights in S3. When users use this patent, they still use the traditional method to extract keywords, but the keyword TF-IDF is the corrected vocabulary importance weight. In order to facilitate the explanation of this patent, this patent will record the continuously modified and corrected TF-IDF as Importance_Score. This patent aims to propose a systematic technical solution for continuously modifying and correcting TF-IDF, so that the rationality of the keyword importance weight is continuously improved in the offline environment, and the vocabulary is expanded at the same time. The inference end of the production environment adopts a dictionary-based keyword extraction method to achieve fast speed and keyword extraction capabilities combined with the capabilities of a large language model. The specific implementation steps of this patent are as follows: Step 1: Modify the TF-IDF value range: The maximum value of all words in S1 is recorded as max, and the minimum value is recorded as min, and then linearly mapped to 5-100 points. The mapping method is as follows: Thus, all TF-IDF values ​​are mapped to 5 to 100 points.

[0016] In order to illustrate the specific implementation method of this patent, this patent sets Tf-idf with a total of 4 keywords For step 1, max=25, min=17, substitute the formula into the list of keyword importance weights: Step 2: Traverse each text in the pre-training dataset, first extract keywords through the large language model, and then extract keywords based on Tf-idf. The specific example operation is as follows: If a text is Text, then add the prompt command as: You are an algorithm engineer with many years of experience in natural language processing. Please extract important keywords based on the following text content. Please do not output additional information. \n{Text}. For example, the following text content: As an important driving force for the new round of scientific and technological revolution and industrial transformation, artificial intelligence has become an important engine for my country's economic development. In the future, my country will continue to increase its efforts in the research and development and application of artificial intelligence technology, promote high-quality development of the industry, and contribute to the comprehensive construction of a modern country.

[0017] The text content of the keyword extraction task finally input into the open source large language model is: You are an algorithm engineer with many years of experience in natural language processing. Please extract important keywords based on the text content below. Please do not output additional information! As an important driving force for the new round of scientific and technological revolution and industrial transformation, artificial intelligence has become an important engine for my country's economic development. In the future, my country will continue to increase its efforts in the research and development and application of artificial intelligence technology, promote high-quality development of the industry, and contribute to the comprehensive construction of a modern country.

[0018] For the above-mentioned large model input, the open source large language model can usually extract keywords. However, the large language model is usually a generative model. Usually, the large language model usually has redundant outputs, and then through regular matching, such as 2 to 5 words between two commas, etc., the keywords are obtained, thereby obtaining a keyword vocabulary based on the large language model, recorded as Keywords_LLM. If the regular matching cannot be parsed, the text will not be operated on later. For the above text extraction, if the keyword is Keywords_LLM = [artificial intelligence, natural language processing, comprehensive construction], if the text can be parsed through the large language model + regular matching to obtain the keyword list, that is, Keywords_LLM, then the text needs to be extracted through TF-idf. TF-idf keyword extraction is a very mature technology in the industry, that is, the larger the keyword TF-idf value (corresponding to the larger Importance-Score above), the easier it is for the vocabulary to be recorded as the keyword of the text. This patent defaults to recording the TF-idf extracted keyword list as Keywords_Statistics. Based on the above keyword importance scores, the keyword list is obtained as follows: Keywords_Statistics = [Natural language processing, artificial intelligence, economic development, high-quality development].

[0019] Step 3: For Keywords_Statistics and Keywords_LLM of each text in the training set, modify the Importance-Score. The specific operation is as follows: If the vocabulary in Keywords_LLM also appears in Keywords_Statistics, then the importance score of the vocabulary is Importance_Score*1.1, that is, the weight of the above-mentioned "artificial intelligence" and "natural language processing" words is multiplied by 1.1. If the vocabulary in Keywords_LLM does not appear in Keywords_Statistics, then the vocabulary is assigned Importance_Score=5 points, that is, "comprehensive construction" is 5.

[0020] That is to say That is, it has the lowest importance in the vocabulary importance list. If the keyword also appears in other texts extracted by the big model in the future, that is, the above-mentioned "comprehensive construction", the importance of the keyword will continue to increase according to *1.1, that is, it will continue to integrate the semantic understanding ability of the big model. Through this operation, the vocabulary expansion is completed. At the same time, as the number of times the big model extracts keywords for this vocabulary, the importance score will gradually increase. If the vocabulary in Keywords_Statistics does not appear in Keywords_LLM, the importance of the vocabulary remains unchanged. (In Step 3, the extraction of keywords for pre-trained Chinese corpus still depends on the offline inference speed of the big model. This patent recommends inferring 100,000 texts. Through the same operation of 100,000 texts, the importance scores of all vocabulary can be adjusted, and the vocabulary expansion is completed based on the big language model.) When encountering new industry field texts, they can be continuously adjusted and updated.

[0021] Step 4: The maximum importance of all words after adjustment is recorded as Score_max, and the minimum importance is recorded as Score_min, which is linearly mapped to 0-100 points. The mapping method is as follows: Step 5: Map the Importance_Score to 0 to 100 points for keyword extraction: Importance_Score is equivalent to TF-IDF to extract keywords, thus completing the keyword capability based on this patent.

[0022] The large language model extracts keywords through prompt instructions. The extracted keyword vocabulary has the ability to understand, but the format of the extracted keyword results is not stable enough under low probability conditions, and there will be redundant output. At the same time, the speed is low when deployed to customers in the production environment. The keyword extraction speed based on Tf-idf is fast, but due to the lack of semantic understanding ability, new words cannot be extracted. At the same time, Tf-idf is a statistical indicator and lacks semantic understanding ability. This patent aims to let the big model play a behind-the-scenes role. Through the big language model on the basis of TF-IDF, the implementation method of this patent continuously expands the vocabulary. At the same time, with the help of the semantic understanding ability of the big model, the TF-IDF value is adjusted, so that important words are extracted as keywords by the big language model many times, and their importance continues to increase. The online environment uses TF-idf to extract keywords at a fast speed. That is, compared with the current technology, this patent makes full use of the big model technology and TF-idf technology, and proposes a keyword extraction capability that makes up for its shortcomings. While continuously completing the vocabulary expansion through the big model, the word weight is adjusted at the same time, and the TF-idf weight value is given semantic characteristics. Therefore, the keyword extraction effect is fast and the semantic understanding ability of the big language model is integrated.

[0023] This patent solution can be implemented. Under the premise that the software and hardware facilities fully meet the conditions required for the operation of the deep learning model, the steps described in the patent can be fully followed to finally obtain the final result described in this patent. The difficulty of modifying the solution described in this patent is moderate. After having an understanding of natural language processing knowledge and a clear understanding of the idea of ​​the method described in this patent, modifications can be made, tested and verified. Application scenarios of this invention: Intelligent question and answer: In the knowledge question and answer based on RAG, the knowledge base obtained through semantic search can use keyword extraction technology to match the knowledge base with keywords for secondary search and sorting, filter out more irrelevant knowledge capabilities, realize the controllability of RAG knowledge question and answer, and improve the effect of RAG. Separate ability of keyword extraction: This patent can be used as a separate ability to facilitate users to quickly determine the theme to be expressed in the text.

[0024] Glossary: Large language models: Large language models are usually built based on GPT-like architectures, with parameters usually exceeding 1 billion. They are pre-trained on large-scale text data, learn the semantics and context of the language, and can generate coherent and reasonable text.

[0025] Keyword extraction: Keyword extraction aims to identify and extract the most representative keywords or phrases that best reflect the text theme from the text. Common methods include word frequency, TF-IDF algorithm, TextRank algorithm, etc. These methods can automatically extract key information from the text and provide support for tasks such as text understanding and information retrieval.

[0026] TF-IDF: TF-IDF is a commonly used weighting technique for information retrieval and text mining, used to evaluate the importance of a word to a set of documents or a corpus. TF stands for word frequency, which measures how often a word appears in a document. The more times a word appears in a document, the more important it is. IDF (inverse document frequency) measures how often a word appears in the entire corpus. If a word appears in many documents, its IDF will be low, indicating that it is a common word and not important. On the contrary, if a word only appears in a few documents, its IDF will be very high, indicating that it has a high importance. The calculation formula of TF-IDF is TF multiplied by IDF, which can get the overall importance of a word in a document. By weighting all the words in the document with TF-IDF, the importance score of each word can be obtained, so as to perform tasks such as keyword extraction, information retrieval, and RAG question answering.

Claims

1. A keyword extraction method based on the understanding ability of a large language model, Features The steps include: S1. Create a large vocabulary Collect relevant words from natural language processing word segmentation tools such as Jieba, summarize them into the list words, and remove duplicate words; S2. Collect and summarize TF-IDF text corpus TF-IDF is produced using a large language model pre-training dataset, and TeleChat pre-training dataset TeleChat-PTD is used; S3. Create the TF-IDF value for each word in S1 The TF value is taken as the average value method, and the words in any specific text in the large language model pre-training dataset are recorded as , the normalized word frequency is expressed as: In the above formula yes In a training corpus The number of occurrences in the denominator In this training corpus The sum of the length of characters; through a text will get a file a word , count the number of times the word appears in all pre-training corpus texts , take the average value of all pre-training corpus texts as the TF value of the word, and get the TF value of all words through the above method. The TF value is written as uppercase ; Inverse Document Frequency (IDF) is a measure of the general importance of a word. The total number of texts in the training set is divided by the number of training texts containing the word, and the logarithm of the quotient is obtained: : The total number of training texts in the pre-training corpus : Contains words The number of files is to prevent the divisor from being zero, so it is generally used The TF-IDF value of each word can be obtained through the above method; S4. Create vocabulary importance_score value The TF-IDF value of each word represents the initial value of the word's importance weight, and the keyword extraction capability can be completed based on the TF-IDF value of each word. However, the words collected in step S1 are corrected in their TF-IDF values ​​and new words are expanded.

2. According to claim 1, a keyword extraction method based on the understanding ability of a large language model, Features The S4 specifically includes the following steps: step1 The TF-IDF value range is modified. The maximum value of all words in S1 is recorded as max, and the minimum value is recorded as min. Then it is linearly mapped to 5-100 points. The mapping method is as follows: Thus, all TF-IDF values ​​are mapped to 5 to 100 points; Step 2 Traverse each text in the pre-trained dataset, first extract keywords through the large language model, and then extract keywords based on Tf-idf. For the above large model input, the open source large language model extracts keywords. However, the large language model is usually a generative model, and usually has redundant outputs. Then, through regular matching, obtain the keyword vocabulary based on the large language model, recorded as Keywords_LLM; if the regular matching cannot be parsed, the text will not be processed later; Step 3 Modify the Importance-Score for Keywords_Statistics and Keywords_LLM of each text in the training set; continuously integrate the semantic understanding ability of the big model to complete the expansion of the vocabulary. At the same time, as the number of times the big model extracts keywords, the importance score will gradually increase; Step 4 The maximum importance of all words after adjustment is recorded as Score_max, and the minimum importance is recorded as Score_min, which is linearly mapped to 0-100 points. The mapping method is as follows: Step 5 Map to Importance_Score from 0 to 100 for keyword extraction, and equate Importance_Score to TF-IDF to extract keywords.

Images