Data set keyword generation and screening method based on large language model

Abstract

The invention provides a data set keyword generation and screening method based on a large language model, and belongs to the field of natural language processing and artificial intelligence. The method comprises the following steps: collecting comment text data and known keywords thereof; dividing the text data into blocks; generating preliminary keywords by the large language model; performing preliminary screening on the preliminary keywords in the preliminary keyword list in combination with a TF-IDF algorithm and a TextRank algorithm; and performing semantic and emotion correlation screening on the preliminarily screened keywords through a large language model to obtain final keywords. According to the method, the keyword generation and screening technology is remarkably improved, and the method is particularly suitable for large-scale text analysis tasks needing high precision and high efficiency.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and artificial intelligence, and in particular to a method for generating and screening keywords of a data set based on a large language model. Background Art

[0002] Internet platforms, such as Zhihu, Weibo, and Douyin, generate hundreds of millions of user comments every day. For these large-scale comment text data, how to effectively generate and expand keywords is a crucial issue. For example, in the analysis of legislative feedback, accurately extracting keywords related to legislative opinions from a massive corpus is of great significance for improving the efficiency of information processing and the accuracy of decision-making. The generation and expansion of keywords is one of the key steps in text analysis. Through high-accuracy keyword extraction, the ability to obtain important information from a large-scale corpus can be effectively improved.

[0003] In the prior art, there are some keyword generation solutions based on large-scale pre-trained language models. By pre-training on a wide range of text data, these models are able to understand complex language patterns and generate keywords related to the input text. However, existing solutions usually rely on one-time generation and lack subsequent keyword screening steps, resulting in the generated keyword set containing irrelevant or redundant words, which affects the actual application effect.

[0004] Traditional methods such as TF-IDF and word vector models (such as Word2Vec) perform poorly in dealing with semantic ambiguity and polysemy in complex texts. Especially in comment texts, which are mostly short texts, these methods cannot effectively capture deep semantic relationships, and the generated keyword sets often lack high semantic relevance and practical application value, making it difficult to reflect the core content of the text.

[0005] For domain-specific terms (such as legal terms), the keywords generated by existing large language models may deviate from actual needs and contain some irrelevant words. In the field of legislation, the quality requirements for keyword generation are higher and must have high accuracy and legal relevance to support legal analysis and decision-making, but existing models often have difficulty meeting these requirements.

[0006] The effectiveness of keywords depends on whether they can accurately reflect the topic and sentiment of the text. Existing large language models usually have difficulty processing contextual information in multi-level text environments when generating keywords, which may result in the generated keywords not accurately reflecting the true meaning of the text.

[0007] When processing large-scale review text data, existing technologies (such as LDA models and deep learning models) are difficult to meet the needs of rapid response due to high model complexity and high computing resource requirements. These technologies often require a lot of computing resources and time when processing large-scale text data, making it difficult to achieve real-time processing, limiting their efficiency and feasibility in practical applications. Summary of the invention

[0008] In order to solve the problems in the prior art, the present invention proposes a data set keyword generation and screening method based on a large language model.

[0009] The present invention adopts the following technical solutions:

[0010] A method for generating and screening keywords for a data set based on a large language model, comprising:

[0011] 1) Collect comment text data on a specific topic on the Internet, pre-process the comment text data, and obtain known keywords from the comment text data;

[0012] 2) Segment the preprocessed text data into multiple text blocks, where the length of each text block does not exceed the maximum input limit of the large language model;

[0013] 3) For each text block, multiple candidate keywords related to the text block content and known keywords are generated through the large language model; the candidate keywords of each text block are integrated to obtain a preliminary keyword list;

[0014] 4) Preliminary screening of the preliminary keywords in the preliminary keyword list is performed by combining the TF-IDF algorithm and the TextRank algorithm to obtain the keywords after preliminary screening;

[0015] 5) The keywords after preliminary screening are screened for semantic and sentiment relevance through a large language model to obtain the final keywords.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. The accuracy and relevance of keyword generation are significantly improved: This invention combines the powerful semantic understanding ability of large language models (such as GPT-4) and traditional TF-IDF and TextRank algorithms, and optimizes the screening results with a weighted mechanism. Through multi-level screening and optimization, the accuracy and relevance of keyword generation are greatly improved. Especially in text processing in specific fields, the generated keywords can not only accurately reflect the core content of the text, but also avoid the interference of redundant and irrelevant words.

[0018] 2. Improved semantic understanding and context dependency processing capabilities: When processing complex legislative texts, existing methods (such as TF-IDF, Word2Vec, etc.) have significant deficiencies in capturing semantic depth and contextual relationships. The large language model, trained on a large corpus and with a deeper architecture, has stronger context understanding and semantic analysis capabilities, and can accurately identify important semantics in the text, thereby generating keywords with higher semantic consistency. This improvement not only improves the quality of generated keywords, but also enhances adaptability to complex commentary texts.

[0019] 3. Reduce data dependency and make up for the shortcomings of legal text corpora: In legal text processing scenarios, high-quality annotated data suitable for semantic relevance judgment is usually lacking, which restricts the effectiveness of traditional methods. However, large language models have been extensively pre-trained on a large number of general and specialized corpora, covering rich semantic information and able to make up for the shortcomings of commentary text data. The present invention utilizes these trained large language models, without relying on large amounts of specialized annotated data, to maintain efficient semantic relevance judgments in commentary text scenarios, thereby significantly reducing the system’s training and data acquisition costs.

[0020] 4. Optimization of sentiment and elimination of semantic repetition: The present invention further uses a large language model to perform semantic and sentiment analysis on the generated keywords, screen out semantically or sentimentally repetitive words, and retain keywords with more generalization and sentiment. This optimization step makes the generated keywords more semantically consistent and more accurate in sentiment expression, especially in legal texts, and can better reflect the sentiment and core content of the text.

[0021] 5. Significantly improved processing efficiency: Optimize computing efficiency through the high responsiveness of large language models: Traditional keyword judgment methods (such as BERT-based models) often require a lot of computing resources and a long processing time in practical applications, especially when facing large-scale text or real-time processing requirements. The present invention integrates multiple trained large language models (such as GPT-4). These models can be directly applied to keyword generation tasks due to their efficient reasoning ability and fast response characteristics, thereby significantly improving the system's computing efficiency and reducing reasoning delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure is a flow chart of the keyword generation and screening method of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0024] In order to solve the problems of accuracy, relevance, semantic understanding, context dependency, processing efficiency and other issues existing in the prior art in keyword generation and screening, the present invention proposes a keyword generation and screening method based on a large language model, which is suitable for large-scale text analysis of online comments. The present invention has made significant progress in keyword generation and screening technology, and is particularly suitable for large-scale text analysis tasks that require high precision and high efficiency. This embodiment collects comment texts centered on the topic of employment, and screens them using a keyword generation set. The resulting keywords help researchers and others understand the public's opinions and emotional tendencies on current employment, thereby further implementing more reasonable and popular policy adjustments and other measures. The following is a detailed explanation of the technical solution:

[0025] (1) Acquisition and preprocessing of text data: For a specific topic, use the keywords representing the topic to search, collect the comment texts of users on various Internet platforms, and form comment text data. The keywords used in the search are known keywords. The comment texts are public comment content searched on the Internet platform and do not contain the personal information of the commentator.

[0026] Before keyword generation, the review text data needs to be preprocessed. The preprocessing process includes data cleaning, deduplication, format normalization and other steps to ensure the quality and consistency of the data input into the model.

[0027] (1.1) Data reading: Use the Python library pandas to read raw text data and convert it into a string format that can be used for subsequent processing.

[0028] (1.2) Data cleaning: Use tools such as regular expressions to remove noise data in the text (such as extra spaces, meaningless symbols, or other interfering content) to improve the quality of the model's input data.

[0029] (1.3) Deduplication: By performing string matching or deduplication based on text similarity algorithms, the input text data is ensured to have no redundant entries, which facilitates subsequent processing.

[0030] As a preferred solution of this embodiment, on the basis of data cleaning, deduplication, and format normalization, a natural language processing (NLP) library (such as NLTK or spaCy) is used for further text preprocessing, such as grammar correction, language detection, etc. Advanced regular expression technology or machine learning models can also be used to detect and repair format errors in input text. A text preprocessing model based on deep learning (such as the Transformer architecture) can also be used to automatically detect and repair noise and format errors in input text. This method can better understand the semantic structure of the text, but may increase computational complexity and processing time.

[0031] (2) Text data segmentation: After text preprocessing, the text data is further segmented into multiple small blocks, each of which does not exceed the maximum input limit of the large language model, because the large language model has strict restrictions on the input length. In order to take into account the richness of all input text data, the text data is further segmented into multiple small blocks. The length of each small block does not exceed the maximum input limit of the large language model. This segmentation technology ensures that the size of each input text is within the processing range of the large language model, which not only avoids the memory overflow problem of the model, but also optimizes the computing efficiency.

[0032] Specifically, a sliding window segmentation method or a natural segmentation based on sentences or paragraphs may be adopted.

[0033] Sliding window segmentation method: In the text segmentation process, the sliding window method can be used to cut the text according to a fixed window size, and there is overlap between adjacent windows. This method can reduce the boundary effect and improve the integrity of the context information retained in the segmentation process, but it will also increase the amount of calculation.

[0034] Natural segmentation based on sentences or paragraphs: If you want to improve semantic understanding, you can use natural language processing tools (such as sentence segmentation algorithms or paragraph segmentation algorithms) to segment based on natural breakpoints to avoid truncation of important sentences or paragraphs. This can be used in scenarios that require high contextual semantic integrity.

[0035] (3) Keyword generation based on large language model: To improve the accuracy and relevance of keyword generation, the present invention uses an advanced large language model (such as GPT4) for keyword generation. This model has strong context understanding and semantic analysis capabilities through large-scale pre-training, and can generate highly relevant keywords.

[0036] Keyword generation process: Construct a prompt for keyword generation for each text block, which includes known keywords and the content of the current text block. Through the design of the prompt, the large language model is guided to generate candidate keywords that are highly relevant to the text content and known keywords. In this embodiment, 5 candidate keywords are generated for each text block. In order to improve the accuracy of the generation, the hyperparameters of the large language model are set to a higher temperature value to reduce randomness, and a higher maximum generation length is set to ensure sufficient keyword generation. Since different text blocks may generate the same or similar keywords, this step also requires deduplication and merging of the keywords generated by all text blocks to form a preliminary keyword list.

[0037] As a preferred solution of this embodiment, keyword generation can use a multi-model fusion method, combining the output results of multiple pre-trained language models, for example, first using the GPT-3 model to generate preliminary keywords, and then using the GPT-4 model to further screen and optimize the keywords. This method takes advantage of the advantages of different models and improves the quality of keyword generation.

[0038] In addition to GPT-4, you can also use pre-trained models based on the Transformer architecture, such as BERT and RoBERTa, to generate keywords. These models have good context understanding capabilities and perform well on some specific tasks, making them suitable as alternatives to keyword generation models.

[0039] By utilizing the powerful semantic analysis capability of the large language model, in this step, the present invention can generate keywords that are highly relevant to the text content, significantly improving the quality of keyword generation.

[0040] (4) Optimization of initial screening of keywords: To solve the quality problem of the prior art in keyword generation, the present invention introduces a secondary keyword screening mechanism. The initially generated keywords may include some broad and irrelevant words. To improve the accuracy and applicability of keywords, the present invention performs multi-level screening.

[0041] Combining the advantages of TF-IDF and TextRank, the preliminary keywords in the preliminary keyword list are screened and optimized. The specific steps are as follows:

[0042] (4.1) Keyword screening based on TF-IDF: All text blocks are vectorized using TfidfVectorizer in the sklearn library, and the TF-IDF value of each preliminary keyword in the entire review text data is calculated based on the TF-IDF algorithm. All generated keywords are sorted according to the calculated TF-IDF value, and the 15 most representative keywords are screened out.

[0043] (4.2) Keyword screening based on TextRank: Segment the review text data and use networkx to build a word co-occurrence network. Use the TextRank algorithm to sort the nodes in the word co-occurrence network and screen out the top 15 preliminary keywords with the highest semantic weight.

[0044] (4.3) Weighted screening and optimization: Assign 50% weight to each of the TF-IDF value and the semantic weight of TextRank, sum the weights of the preliminary keywords screened out by the two methods, calculate the accuracy and relevance of each preliminary keyword, and generate the final score. Sort the preliminary keywords according to the final score, and select the top 10 preliminary keywords with the highest final scores, which are the keywords after preliminary screening.

[0045] As another solution of this embodiment, the screening process of this step can also adopt a screening mechanism based on similarity: calculate the similarity between the candidate keywords and the known keywords (such as cosine similarity or Euclidean distance of word vectors), sort them according to the similarity scores, and screen out the keywords with the highest scores. Although this method is direct and simple, it depends on the quality of the word vectors and the rationality of the similarity measurement.

[0046] In addition to automatic screening, keyword screening can also be combined with manual intervention to ensure that the generated keywords are highly relevant to the text content through manual inspection and adjustment.

[0047] Through the preliminary screening mechanism of keywords, the present invention significantly improves the accuracy and relevance of keywords, ensuring that the generated keywords can accurately reflect the text content.

[0048] (5) Semantic repetition and sentiment relevance screening of keywords: After obtaining the keywords after preliminary screening, in order to further optimize the results, the present invention uses the GPT-4 model to perform semantic and sentiment relevance screening:

[0049] Semantic similarity deduplication: Use the GPT-4 model to perform semantic analysis on the generated keywords. For a group of semantically similar words, choose to keep one and delete the other keywords in the same group.

[0050] Sentiment optimization: For keywords with sentiment, we will give priority to retaining those with sentiment. If there is no sentiment, we will give priority to retaining more general keywords to ensure that the results can accurately reflect the sentiment and theme of the text content.

[0051] The above-mentioned semantic similarity deduplication and sentiment tendency optimization are both performed through the GPT-4 model, including the judgment of semantic similarity and whether there is sentiment tendency.

[0052] The final generated keywords are filtered and optimized and saved in a JSON format file for subsequent analysis.

[0053] Through the above technical solution, the present invention effectively solves the shortcomings of the existing technology in keyword generation and screening, and provides an efficient, accurate and widely applicable solution, which is particularly suitable for large-scale text analysis with clear themes. The design of each step and module takes into account the actual application requirements and the feasibility of technical implementation, providing reliable technical support for the realization of comprehensive text analysis and keyword extraction.

[0054] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for generating and screening keywords in a data set based on a large language model, characterized in that: include: 1) Collect comment text data on specific topics on the Internet and preprocess the comment text data; Get known keywords of comment text data; 2) Segment the preprocessed text data into multiple text blocks, where the length of each text block does not exceed the maximum input limit of the large language model; 3) For each text block, multiple candidate keywords related to the text block content and known keywords are generated through the large language model; the candidate keywords of each text block are integrated to obtain a preliminary keyword list; 4) Preliminarily screening the preliminary keywords in the preliminary keyword list to obtain keywords after preliminary screening; 5) The keywords after preliminary screening are screened for semantic and sentiment relevance through a large language model to obtain the final keywords.

2. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: The preprocessing includes data cleaning and deduplication processing.

3. The method for generating and screening keywords of a data set based on a large language model according to claim 2, characterized in that: The data cleaning comprises: using regular expressions to clean up noise information in the comment text data; The de-duplication process is to detect duplicate texts in the comment text data through string matching or text similarity algorithm, and remove the duplicate texts.

4. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: The known keywords are words that describe the specific topic.

5. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: Step 2) Use sliding window method or semantic segmentation method to segment text blocks.

6. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: The large language model is a GPT4 model, a BERT model, a RoBERTa model or a T5 model.

7. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: The method of generating multiple candidate keywords related to the text block content and known keywords through the large language model specifically includes: inputting the known keywords and the current text block into the large language model, and requiring the large language model to generate several candidate keywords most related to the known keywords based on the current text block content.

8. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: The candidate keywords of each text block are integrated to obtain a preliminary keyword list, specifically: for the candidate keywords of all text blocks, the same candidate keywords are removed and then merged into a preliminary keyword list.

9. The method for generating and screening keywords for a data set based on a large language model according to claim 1, characterized in that: Step 4) is specifically as follows: 4.1) Using the TF-IDF algorithm to calculate the TF-IDF value of each candidate keyword in the comment text data; sorting the preliminary keywords in the preliminary keyword list according to the calculated TF-IDF values, and selecting the m preliminary keywords with the highest TF-IDF values; 4.2) Segment the review text data and construct a word co-occurrence network. A node in the word co-occurrence network represents a word. The nodes in the word co-occurrence network are scored using the TextRank algorithm. The score obtained is the semantic weight of the corresponding word. The preliminary keywords in the preliminary keyword list are sorted according to the semantic weight, and the m preliminary keywords with the highest semantic weight are screened out; 4.3) Assign 50% weight to each of the TF-IDF value and the semantic weight. For the preliminary keywords screened out in steps 4.1) and 4.2), add up their TF-IDF values ​​and semantic weights to calculate the final score. Several preliminary keywords with the highest final scores are selected as keywords after preliminary screening.

10. The method for generating and screening keywords of a data set based on a large language model according to claim 1, characterized in that: Step 5) is specifically: Through the large language model, the semantic similarity analysis is performed on the keywords that have been initially screened. Based on the sentiment tendency optimization principle, one of the keywords obtained from the group of keywords with similar semantics is selected and retained, and the other keywords in the group are deleted. The last keyword retained is the final keyword. The sentiment tendency optimization principle is: give priority to retaining words with sentiment tendencies, and if the keywords in the group have no sentiment tendencies, give priority to retaining more general keywords.

Images