Short text theme recognition method based on comparative learning and LLMs

Abstract

The invention discloses a short text theme recognition method based on comparative learning and LLMs, and relates to the field of short text theme recognition, and the method comprises the steps: firstly splicing sentence vectors of keywords extracted by an original text and LLMs to generate enhanced vector representation; then generating a pseudo-tag pre-training MLP by adopting a clustering algorithm, screening an edge text and a center text based on an MLP clustering result to construct a positive and negative sample pair, introducing supervised contrast learning, and optimizing an embedded model and the MLP; and finally, carrying out anomaly discrimination on the edge sample by calling LLMs, and extracting a cluster topic feature in combination with TF-IDF. In experiments of four text data sets, the model is averagely 9.62% and 4.59% higher than a suboptimal model in theme consistency TC and theme diversity TD, which indicates that the accuracy and robustness of short text theme recognition are remarkably improved through a multi-stage semantic enhancement and contrast learning mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of short text topic identification, in particular to a short text topic identification method based on contrastive learning and LLMs. BACKGROUND

[0002] With the rapid development of the Internet and mobile communication technology, short text data (such as social media posts, news headlines, user reviews, instant messages, etc.) has shown explosive growth. Efficient and accurate identification of the topic of short text is of great significance to public opinion monitoring, user portrait construction, information retrieval and recommendation, content understanding, etc. However, the inherent characteristics of short text topic identification, such as sparse text semantic information, ambiguous topic clusters of edge text, and susceptibility to cluster abnormal text, make it difficult for traditional text topic identification models to achieve ideal results when modeling.

[0003] For example, traditional probabilistic methods such as BTM and its extension LS-BTM, while alleviating information sparsity through word pair co-occurrence statistics, are unable to capture deep semantics due to their static modeling mechanism, resulting in insufficient semantic expression of topics.

[0004] Methods based on word / document vectorization techniques such as Word2Vec, Doc2Vec combined with clustering algorithms are prone to unstable clustering results due to the anisotropy of sentence vector space, exacerbating the ambiguity of edge text attribution.

[0005] Hybrid models such as BERT-LDA, while integrating context vectors and topic distributions, still face the challenges of local optimization and vector space distortion, and lack robustness to noise and abnormal text.

[0006] PromptTopic and other methods based on prompt engineering leverage the advantages of large language models (LLMs) in semantic understanding and zero-shot inference, effectively compensating for the lack of context in short text. However, these methods cannot avoid the inherent hallucination phenomenon of LLMs, which may lead to topic deviation or distorted interpretation. SUMMARY

[0007] The present application aims to provide a short text topic identification method based on contrastive learning and LLMs, which solves the problems of semantic insufficiency, ambiguous topic clusters of edge text, and interference of cluster abnormal text on topic quality in short text topic identification, and improves the accuracy of short text topic identification.

[0008] To achieve the above-mentioned purpose, the present application provides a short text topic identification method based on contrastive learning and LLMs, which comprises the following steps:

[0009] Step 1: collecting short text data, constructing a first original sample set and a second original sample set based on the short text data, the first original sample set comprising a plurality of first original samples, and the second original sample set comprising a plurality of second original samples;

[0010] Step 2: extracting keywords of the plurality of first original samples and the plurality of second original samples respectively by using a large language model to obtain a plurality of first keyword samples and a plurality of second keyword samples;

[0011] Step 3: converting the plurality of first original samples, the plurality of first keyword samples, the plurality of second original samples and the plurality of second keyword samples into a plurality of first original vectors, a plurality of first keyword vectors, a plurality of second original vectors and a plurality of second keyword vectors respectively by using a first sentence embedding model;

[0012] Step 4: concatenating any first original vector and its corresponding first keyword vector to obtain a plurality of first enhanced vectors, and concatenating any second original vector and its corresponding second keyword vector to obtain a plurality of second enhanced vectors, constructing a first enhanced vector set based on the plurality of first enhanced vectors, and constructing a second enhanced vector set based on the plurality of second enhanced vectors;

[0013] Step 5: applying a clustering algorithm to the first enhanced vector set, generating a corresponding pseudo label for any first enhanced vector, and constructing a training set based on all the first enhanced vectors and their corresponding pseudo labels;

[0014] Step 6: constructing a first multilayer perceptron model, performing initial training on the first multilayer perceptron model based on the training set, and obtaining a second multilayer perceptron model;

[0015] Step 7: inputting the second enhanced vector set into the second multilayer perceptron model to output clustering results of the second enhanced vector set;

[0016] Step 8: selecting edge texts and center texts in the second original sample set from the clustering results, and constructing a set of positive and negative sample pairs based on the edge texts and the center texts;

[0017] Step 9: training the second multilayer perceptron model based on the set of positive and negative sample pairs to obtain a third multilayer perceptron model, and adjusting the first sentence embedding model based on the set of positive and negative sample pairs to obtain a second sentence embedding model;

[0018] Step 10: identifying the theme of a set of to-be-identified short texts based on the second sentence embedding model and the third multilayer perceptron model to obtain an identification result.

[0019] The present invention divides short text data into a first set of original text samples and a second set of original text samples to facilitate subsequent model training and optimization. Traditional text clustering relies on similarity metrics based on raw semantic embeddings, but this is susceptible to noise interference in the semantically ambiguous context of short text. Therefore, the present invention accesses a large language model to extract task-oriented first and second keyword samples to explicitly enhance the semantic discriminability of the text. A first sentence embedding model is then used to perform sentence embedding on the original text samples and keyword samples, respectively. The vectors of these two are then concatenated for subsequent clustering and comparative learning. The multilayer perceptron (MLP) model offers significant advantages in nonlinear modeling and efficient inference. Its forward propagation directly outputs class probabilities, avoiding the overhead of repeated iterations and improving inference efficiency. Therefore, the present invention selects the MLP for clustering output. To prevent excessive bias in the MLP's initial clustering predictions, the present invention applies a clustering algorithm to the first enhanced vector set and generates pseudo-labels for these vectors to construct a training set, providing a reliable training foundation for the first multilayer perceptron model. After the training is completed, a second multi-layer perceptron model is obtained. However, the second multi-layer perceptron model may not be able to fully open the subtle boundaries between different clusters in the high-dimensional semantic space, and the topic clustering cluster to which the edge text belongs is fuzzy. Therefore, the present invention further introduces a supervised contrastive learning mechanism, specifically: first, the clustering result of the second enhanced vector set is output by the second multi-layer perceptron model, and then the edge text and the center text are selected from the clustering result to construct a positive and negative sample pair set. Finally, based on the positive and negative sample pair set, the second multi-layer perceptron model and the first sentence embedding model are optimized and adjusted. Under the action of supervised contrastive loss, the cohesion of samples in the same cluster is strengthened, and the model's ability to discriminate the cluster to which the edge text belongs is improved, ultimately achieving the effect of improving the accuracy of short text topic recognition.

[0020] Furthermore, in step 6, the loss function for the initial training of the first multi-layer perceptron model based on the training set is ,in:

[0021] ;

[0022] in, is the total number of samples, is the number of categories, For samples For category The true label, Predict samples for the model For category The loss function measures the difference between the model prediction and the pseudo label, which encourages the model to better fit the category label of each text.

[0023] Further, in step 9, the loss function for training the second multi-layer perception model based on the positive and negative sample pair set is wherein:

[0024] ;

[0025] wherein, is the total number of samples, and are the vector representations of the sample and the sample , respectively, is the square of the Euclidean distance between the vector representations of the sample and the sample , represents whether the text pair belongs to the same category, if the sample and the sample belong to the same category, then , if the sample and the sample do not belong to the same category, then , the loss function improves the similarity of the texts within the cluster by minimizing the distance between the samples in the same cluster, and promotes the clustering effect.

[0026] Further, in step 8, the selection of the edge text and the center text in the second original text sample set from the clustering result comprises:

[0027] calculating the confidence index and the information entropy index of the classification prediction of the second multi-layer perception model;

[0028] calculating a comprehensive index based on the confidence index and the information entropy index;

[0029] selecting the edge text based on the comprehensive index.

[0030] wherein, the edge text is usually located in the boundary area of the class cluster and has high uncertainty, and it contains rich context information that can provide more detailed support for class cluster division. When the edge text is included in the supervised learning process, it can significantly enhance the model's understanding of complex boundaries, thereby optimizing the clustering structure and making the final result more accurate. The present application selects the edge text based on the information entropy index and the confidence index. The information entropy index can quantify the uncertainty of the model in classifying samples, accurately capturing texts with ambiguous semantics or overlapping boundaries between multiple clusters, thereby enhancing the discriminability of edge samples. The confidence index measures the degree of certainty of the model in judging the sample's membership, effectively excluding noisy samples caused by high uncertainty.

[0031] Further, the calculation formula of the confidence index is:

[0032] ;

[0033] wherein, is the confidence index, is the category probability distribution output by the second multi-layer perception model, is a sample the maximum probability of the cluster to which the sample belongs;

[0034] The calculation formula of the information entropy index is:

[0035] ;

[0036] wherein, is the information entropy index, is the information entropy, which measures the uniformity of the probability distribution of the sample, is the number of sample clusters, is the model predicted sample is the probability of the category;

[0037] The calculation formula of the comprehensive index based on the confidence index and the information entropy index is:

[0038] ;

[0039] wherein, is the comprehensive index, is an adjustment coefficient;

[0040] The selecting the edge text based on the comprehensive index comprises:

[0041] If the exceeds a preset index threshold, the sample is marked as the edge text.

[0042] Further, in step 8, the constructing a positive-negative sample pair set based on the edge text and the center text comprises:

[0043] obtaining the cluster center vector in the clustering result;

[0044] combining the second enhanced vector corresponding to the center text and the cluster center vector in the same cluster as the center text as a positive text pair set;

[0045] combining the second enhanced vector corresponding to the edge text and the cluster center vector in the same cluster as the edge text as a negative text pair set.

[0046] ​The negative text pair set is used to strengthen the discrimination ability of the model to cross-cluster instances, through supervised contrast learning, so that the embedding vector of the edge text is far away from the cluster center in the semantic space and closer to the required target cluster center, thereby accurately adjusting the cross-cluster division and improving the class distinction degree.

[0047] Further, the formula for obtaining the cluster center vector in the clustering result is:

[0048] ;

[0049] wherein, is the cluster center vector of the cluster , is the vector representation of the sample , is the predicted probability that the sample belongs to the cluster ; the application obtains the clustering label to which each text sample belongs by using the vector representation of each text sample and the MLP model, and then calculates the center vector of each cluster by using the weighted mean for each cluster.

[0050] Further, in step 9, when the second multi-layer perceptron model is trained based on the positive and negative sample pair set, whether to stop iteration is determined based on the MLP clustering assignment change rate;

[0051] The calculation formula of the clustering assignment change rate is:

[0052] ;

[0053] wherein, is the clustering assignment change rate, is the total number of samples, is the clustering category of the sample in the first iteration, is the clustering category of the sample in the first iteration, is the clustering category of the sample in the first iteration, is a binary indicator function.

[0054] The determination of whether to stop iteration based on the clustering assignment change rate comprises:

[0055] When the clustering assignment change rate is less than a preset change rate threshold, the iteration is stopped.

[0056] Wherein, the determination of whether to stop iteration based on the clustering assignment change rate can effectively control the iteration optimization process of the model, and ensure the convergence and calculation efficiency.

[0057] Furthermore, in step 9, the loss function for adjusting the first sentence embedding model based on the positive and negative sample pairs is ,in:

[0058] ;

[0059] in, is the positive text pair set, is a negative text pair set, is a positive sample pair The cosine similarity of is a negative sample pair The cosine similarity of is a hyperparameter used to adjust the weight of the similarity difference between positive and negative sample pairs. The loss function penalizes sample pairs that violate the sorting constraint through the exponential term, so that the model directly optimizes the relative order of cosine similarity.

[0060] Furthermore, the method further comprises the following steps:

[0061] Step 11: selecting marginal texts in the short text set to be identified from the recognition results as suspected abnormal texts;

[0062] Step 12: Using a large language model to determine the correctness of the classification of the suspected abnormal text, if the classification is incorrect, the suspected abnormal text is marked as abnormal text, and the abnormal text is removed from the set of short texts to be identified, thereby obtaining a set of short texts to be extracted;

[0063] Step 13: Based on the recognition results, the TF-IDF weighting method is used to extract the key words of each cluster in the short text set to be extracted.

[0064] When using the third multi-layer perceptron model to perform topic identification on the short text set to be identified, unstable text (suspected anomalous text) still exists within each cluster. Therefore, the present invention uses a large language model to identify and eliminate these suspected anomalous texts. Finally, the TF-IDF weighted method is used to extract the key words for each cluster in the short text set to be extracted. The TF-IDF method not only considers the frequency of words in the document but also adjusts their distribution across the entire corpus. Through this calculation, it is possible to effectively remove common high-frequency words in the corpus and highlight those that are relatively unique and important to the current cluster.

[0065] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0066] 1. Based on the deep semantic understanding ability of LLMs, this paper extracts text keywords, captures the core semantics of the text, and identifies abnormal text in the cluster after clustering; through supervised comparative learning of edge text and center text, it effectively improves the clustering of fuzzy text and enhances clustering accuracy.

[0067] 2. This method has better consistency when generating keywords and can ensure the diversity between topics. Compared with traditional models, it can better capture the deep semantic information of the text and effectively handle abnormal sample problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation of the embodiments of the present invention;

[0069] Figure 1 It is a flowchart of the short text topic identification method based on contrastive learning and LLMs in the present invention. DETAILED DESCRIPTION

[0070] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0071] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0072] Example 1

[0073] Embodiment 1 of the present invention provides a short text topic identification method based on contrastive learning and LLMs, the method comprising the following steps:

[0074] Step 1: Collect short text data, and construct a first original text sample set and a second original text sample set based on the short text data, wherein the first original text sample set includes a plurality of first original text samples, and the second original text sample set includes a plurality of second original text samples;

[0075] Step 2: Using a large language model, extract keywords from the first original text samples and the second original text samples respectively to obtain a plurality of first keyword samples and a plurality of second keyword samples;

[0076] Step 3: converting the plurality of first original text samples, the plurality of first keyword samples, the plurality of second original text samples, and the plurality of second keyword samples into a plurality of first original text vectors, a plurality of first keyword vectors, a plurality of second original text vectors, and a plurality of second keyword vectors respectively by a first sentence embedding model;

[0077] Step 4: concatenating any of the first original text vectors and its corresponding first keyword vector to obtain a plurality of first enhanced vectors, and concatenating any of the second original text vectors and its corresponding second keyword vector to obtain a plurality of second enhanced vectors, constructing a first enhanced vector set based on the plurality of first enhanced vectors, and constructing a second enhanced vector set based on the plurality of second enhanced vectors;

[0078] Step 5: applying a clustering algorithm to the first enhanced vector set, generating a corresponding pseudo-label for any of the first enhanced vectors, and constructing a training set based on all the first enhanced vectors and their corresponding pseudo-labels;

[0079] Step 6: constructing a first multi-layer perceptron model, performing initial training on the first multi-layer perceptron model based on the training set, and obtaining a second multi-layer perceptron model;

[0080] Step 7: inputting the second enhanced vector set into the second multi-layer perceptron model, and outputting the clustering results of the second enhanced vector set;

[0081] Step 8: selecting edge texts and center texts from the second original text sample set from the clustering results, and constructing a set of positive and negative sample pairs based on the edge texts and the center texts;

[0082] Step 9: training the second multi-layer perceptron model based on the set of positive and negative sample pairs, obtaining a third multi-layer perceptron model, adjusting the first sentence embedding model based on the set of positive and negative sample pairs, and obtaining a second sentence embedding model;

[0083] Step 10: identifying the theme of the set of short text to be identified based on the second sentence embedding model and the third multi-layer perceptron model, and obtaining an identification result.

[0084] In step 1 of the embodiment, the source of short text data is as follows:

[0085] StackOverflow: Kaggle platform released program question and answer challenge, containing about 20000 short question texts, covering 20 technical tags of computer science fields such as Qt, Matlab, Spring, Bash, etc.;

[0086] AgNews: A subset of news headlines, containing about 8000 news headlines, including "World", "Sports", "Business", "Sci / Tech" four topic categories;

[0087] Biomedical: A professional subset based on PubMed literature data screening, covering 20 main research directions in the field of medicine;

[0088] SearchSnippets: Query snippet dataset from Google search, containing 12340 non-consecutive samples composed of only keywords, divided into 8 categories.

[0089] Among them, in step 2, when extracting keywords, those skilled in the art can input instructions according to actual conditions, for example:

[0090] The task setting style is: "short text processing, parsing the center keywords of each sentence".

[0091] The request text style is: "Please parse the keywords of each sentence in the English text separated by '||' to enhance text clustering; please only output the required overall text, and the parsed keywords of different sentences are separated by '||'. Please strictly follow the output format, and parse the text content as follows: {text}". For the text of the request processing, it will be spliced as a processing text with special symbols "||" according to the batch size.

[0092] The above example requests to splice 100 texts per batch according to the request mode, and after obtaining the data returned by the large language model API, the data is divided according to "||".

[0093] Among them, in step 3, the first sentence embedding model can choose the SBERT sentence embedding model of all-MiniLM-L6-v2, which maps the text to a 384-dimensional vector.

[0094] Among them, in step 5, the clustering algorithm is preferably the HDBSCAN clustering algorithm. HDBSCAN is a density-based clustering algorithm that can automatically identify different text topics in the data without pre-setting the number of clusters, and has high robustness in handling noise points and edge samples. In addition, there may be clustering collapse and data sparseness in high-dimensional space clustering, so the UMAP algorithm is used to reduce the dimension of the sentence vector first, and then perform HDBSCAN clustering.

[0095] In step 6, the first multi-layer perception model includes an input layer, a hidden layer and an output layer, the input layer receives a 716-dimensional vector, the hidden layer adopts a double-layer full connection structure, each layer contains 128 neurons, and uses a ReLU activation function to extract high-order semantic features through a nonlinear mapping to avoid the gradient disappearance problem caused by too deep network. The output layer converts the output of the hidden layer into a category probability distribution through a Softmax function, and the output dimension is the same as the preset cluster number. During training, an Adam optimizer is used, the learning rate is set to 1e-3, and the training iteration is set to 20.

[0096] In step 6, the loss function for training the first multi-layer perception model based on the training set is , wherein:

[0097] ;

[0098] , wherein: is the total number of samples, is the number of categories, is the true label of the sample of the category , is the probability of the sample of the category predicted by the model.

[0099] In step 9, the loss function for training the second multi-layer perception model based on the positive and negative sample pair set is , wherein:

[0100] ;

[0101] , wherein: is the total number of samples, and are the vector representations of the sample and the sample , is the square of the Euclidean distance between the vector representations of the sample and the sample , represents whether the text pair is of the same category, if the sample and the sample belong to the same category, then , if the sample and the sample do not belong to the same category, then . In this embodiment, when training the second multi-layer perception model, an Adam optimizer is used, and the learning rate is set to 2e-5.

[0102] In step 8, the edge text and the center text in the second original sample set are selected from the clustering result, comprising:

[0103] The confidence index and the information entropy index of the second multi-layer perception model classification prediction are calculated;

[0104] The comprehensive index is calculated based on the confidence index and the information entropy index;

[0105] The edge text is selected based on the comprehensive index.

[0106] The calculation formula of the confidence index is:

[0107] ;

[0108] Wherein, is the confidence index, is the category probability distribution output by the second multi-layer perception model, is the maximum probability of the sample belonging to the cluster;

[0109] The calculation formula of the information entropy index is:

[0110] ;

[0111] Wherein, is the information entropy index, is the information entropy, is the number of sample clusters, is the probability of the model predicted sample belonging to the category ;

[0112] The calculation formula of the comprehensive index calculated based on the confidence index and the information entropy index is:

[0113] ;

[0114] Wherein, is the comprehensive index, is the adjustment coefficient;

[0115] The edge text is selected based on the comprehensive index, comprising:

[0116] If the comprehensive index exceeds the preset index threshold, the sample is marked as the edge text.

[0117] The adjustment coefficient and the preset index threshold are set by a person skilled in the art according to actual conditions, and can also be screened through experiments, and the present application does not limit this. In the embodiment, the adjustment coefficient is preferably 0.3 and the preset index threshold is preferably 0.4 after screening through experiments.

[0118] In step 8, the set of positive and negative sample pairs is constructed based on the edge text and the center text, including:

[0119] Obtaining a cluster center vector in the clustering result;

[0120] Combining the second enhanced vector corresponding to the center text and the cluster center vector in the same cluster as the center text as a set of positive text pairs;

[0121] Combining the second enhanced vector corresponding to the edge text and the cluster center vector in the same cluster as the edge text as a set of negative text pairs.

[0122] The formula for obtaining the cluster center vector in the clustering result is:

[0123] ;

[0124] Wherein, is the cluster center vector of cluster , is the vector representation of sample , is the prediction probability of sample belonging to cluster .

[0125] In step 8, the edge text and the center text in the second original sample set are selected from the clustering result, and further including:

[0126] Calculate the cosine similarity of each sample and its cluster center vector, arrange all samples in descending order of similarity and select the top most representative center text, and pair with the cluster center vector to form a positive sample pair, wherein the number of negative sample pairs is consistent.

[0127] In step 9, when the second multilayer perception machine model is trained based on the set of positive and negative sample pairs, whether to stop iteration is determined based on a clustering distribution change rate;

[0128] The calculation formula of the clustering distribution change rate is:

[0129] ;

[0130] Wherein, is the clustering distribution change rate, is the total number of samples, For samples In the The clustering categories of the iterations, For samples In the The clustering categories of the iterations, is a binary indicator function;

[0131] The determining whether to stop iteration based on the cluster allocation change rate includes:

[0132] When the cluster allocation change rate is less than a preset change rate threshold, the iteration is stopped.

[0133] Among them, the preset change rate threshold can be set by those skilled in the art according to actual conditions, and the present invention does not limit it. In this embodiment, it is preferably 0.01. In addition, the training process is also subject to the maximum number of iterations. To prevent the algorithm from running infinitely, when the number of learning times is greater than In this embodiment, the maximum number of iterations is set to 100.

[0134] Among them, in step 9, the loss function for adjusting the first sentence embedding model based on the positive and negative sample pairs is ,in:

[0135] ;

[0136] in, is the positive text pair set, is a negative text pair set, is a positive sample pair The cosine similarity of is a negative sample pair The cosine similarity of is a hyperparameter, and in this embodiment, the hyperparameter is set to 20.

[0137] Example 2

[0138] Based on the above embodiment, the method further includes the following steps:

[0139] Step 11: selecting marginal texts in the short text set to be identified from the recognition results as suspected abnormal texts;

[0140] Step 12: Using a large language model to determine the correctness of the classification of the suspected abnormal text, if the classification is incorrect, the suspected abnormal text is marked as abnormal text, and the abnormal text is removed from the set of short texts to be identified, thereby obtaining a set of short texts to be extracted;

[0141] Step 13: Based on the identification result, the topic words of each cluster in the short text set to be extracted are extracted using the TF-IDF weighting method.

[0142] In step 11, the method for selecting the edge text in the short text set to be identified from the identification result is the same as step 8 of embodiment one.

[0143] In step 12, when using a large language model to determine the classification correctness of the suspected abnormal text:

[0144] The point closest to the center vector of each cluster in the embedding space is used as the center text of the cluster, and a text pair set M of suspected abnormal text and its center text is constructed;

[0145] The model task setting style in the API request is: "text pair analysis, specific analysis of whether the text pair belongs to the same topic". The request text style is: "please judge whether each text pair in the text in the form of (t1, t2) belongs to the same topic text, 1 for the same topic, -1 for different topics; please concatenate the judgment results of each text pair with '||', and return the overall answer. Please strictly follow the output format requirements, and analyze the text content as follows: {text}". Among them, the request processing text is composed of text pairs in the set M in the form of "(t1, t2)".

[0146] In this embodiment, the above request is requested in batches of 50 text pairs, until all suspected abnormal texts are processed.

[0147] In step 13, all texts in the short text set to be extracted are merged into a similar document, the weight of each word in the cluster is calculated using the TF-IDF method, and the key words with high weight are selected as the topic words of the short text set to be extracted.

[0148] Embodiment three

[0149] On the basis of the above embodiments, the method of the present application is used to identify the topic of short text, and is compared and analyzed with five baseline models, and the baseline models are as follows:

[0150] LDA: A probabilistic generative model designed to reveal the semantic structure of a text corpus by learning the distribution of latent topics in documents and the distribution of words under topics. (Reference: Wang X, McCallum A. Topics over time: a non-markov continuous-time model of topical trends [C] Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 2006: 424-433.)

[0151] BTM: This model learns the generation probability of word pairs in each topic and the overall distribution characteristics of topics in the corpus by modeling all word pairs in the entire corpus. (Reference: Cheng X, Yan X, Lan Y, et al. BTM: Topic Modeling over Short Texts [J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26: 2928-2941)

[0152] Top2Vec: Utilizes default sentence editor for document embedding, dimensionality reduction and density clustering to automatically identify topics, defines the centroid of document clusters as topic vectors, and finds the closest K topics in the word space using the nearest neighbor. (Reference: Angelov D. Top2Vec: Distributed Representations of Topics [J]. 2020, arXiv Preprint, arXiv: cs.DL / 09470 (2208).)

[0153] BERTopic: A modular topic modeling framework based on pre-trained Transformer models that combines document embedding, dimensionality reduction and density clustering to identify document clusters, and extracts topics within clusters through TF-IDF. (Reference: Grootendorst M. BERTopic: Neural Topic Modeling with A Class-based TF-IDF procedure [J]. 2022, arXiv Preprint, arXiv: cs.DL / 05794 (2203).)

[0154] CSCK: Sentence embedding models pre-trained with CoSENT are used to obtain sentence vectors, clustering is performed using Canopy + improved K-means, and cluster topics are extracted based on TF-IDF. (Reference to prior art: Chen Jianfei, Bu Fanliang, Wang Yifan. Topic identification of offensive comment text based on CoSENT and improved K-means [J]. Science, Technology and Engineering, 2024, 24(31): 13442-13449.)

[0155] This example uses two metrics to measure model performance. The first is the topic consistency metric (TC), which measures the co-occurrence of keywords in the same cluster using normalized pointwise mutual information (NPMI). This approach eliminates scale bias caused by differences in word frequency. The calculation formula for TC is as follows:

[0156] ;

[0157] in, , Expressive words and words The probability of co-occurrence in the same document, and is the probability of each occurring individually. is the total number of topics, For each topic Number of keywords. The value range is [−1, 1]. The higher the value, the more frequently the word pairs in the topic appear in the corpus and the stronger the internal semantic association.

[0158] The second item is the topic diversity indicator (TD), which is used to measure the difference in vocabulary distribution between different topics. It reflects the model's ability to capture diverse topics. The calculation formula for TD is as follows:

[0159] ;

[0160] in, For the theme Before The number of terms after deduplication in the keywords, is the total number of topics. TD falls between (0, 1]. When TD is close to 1, it means that the high-frequency words of each topic have almost no overlap and the topic boundaries are clear. When the TD value is low, it means that there is a large amount of shared vocabulary between different topics, and the topics generated by the model may be confused with each other.

[0161] The final analysis results are shown in Table 1 below:

[0162] Table 1 Performance comparison of different models (methods)

[0163]

[0164] According to Table 1, the method of the present application obtains the highest TC and TD scores on the four datasets. On the StackOverflow dataset, TC reaches 32.78%, which is 5.81% higher than 26.97% of the suboptimal model CSCK, and TD score is 75.62%. On the AgNews dataset, TC and TD are 45.75% and 93.00%, respectively. On the Biomedical dataset, TC is 36.60% and TD is 86.63%. On the SearchSnippets dataset, TC is 32.85% and TD is 92.50%. Overall, the present method exceeds the suboptimal model by 9.62% and 4.59% on average in terms of TC and TD.

[0165] While the preferred embodiments of the application have been described, additional modifications and changes can occur to those skilled in the art once they learn of the basic creative principles disclosed herein. Therefore, the present application is to be interpreted in the broadest and intended sense, as is in accord with the principles and the novel features disclosed herein.

[0166] Obviously, various modifications and changes are possible in the present application without deviating from the scope of the application. It is therefore intended that the present application encompass all such modifications and changes as fall within the scope of the claims and their equivalents.

Claims

1. A short text topic identification method based on contrastive learning and LLMs, characterized by: The method comprises the following steps: Step 1: Collect short text data, and construct a first original text sample set and a second original text sample set based on the short text data, wherein the first original text sample set includes a plurality of first original text samples, and the second original text sample set includes a plurality of second original text samples; Step 2: Using a large language model, extract keywords from the first original text samples and the second original text samples respectively to obtain a plurality of first keyword samples and a plurality of second keyword samples; Step 3: Convert the first original text samples, the first keyword samples, the second original text samples, and the second keyword samples into a plurality of first original text vectors, a plurality of first keyword vectors, a plurality of second original text vectors, and a plurality of second keyword vectors respectively through the first sentence embedding model; Step 4: Concatenate any one of the first original text vectors with its corresponding first keyword vector to obtain a plurality of first enhanced vectors, concatenate any one of the second original text vectors with its corresponding second keyword vector to obtain a plurality of second enhanced vectors, construct a first enhanced vector set based on the plurality of first enhanced vectors, and construct a second enhanced vector set based on the plurality of second enhanced vectors; Step 5: Apply a clustering algorithm to the first enhanced vector set, generate a corresponding pseudo label for any first enhanced vector, and construct a training set based on all the first enhanced vectors and their corresponding pseudo labels; Step 6: constructing a first multi-layer perceptron model, and performing initial training on the first multi-layer perceptron model based on the training set to obtain a second multi-layer perceptron model; Step 7: Input the second enhanced vector set into the second multi-layer perceptron model, and output the clustering result of the second enhanced vector set; Step 8: Selecting marginal texts and central texts from the second original text sample set from the clustering results, and constructing a positive and negative sample pair set based on the marginal texts and the central texts; Step 9: training the second multi-layer perceptron model based on the positive and negative sample pair set to obtain a third multi-layer perceptron model, and adjusting the first sentence embedding model based on the positive and negative sample pair set to obtain a second sentence embedding model; Step 10: Identify the topic of the short text set to be identified based on the second sentence embedding model and the third multi-layer perceptron model to obtain a recognition result.

2. The short text topic identification method based on contrastive learning and LLMs according to claim 1 is characterized in that In step 6, the loss function for training the first multi-layer perceptron model based on the training set is ,in: ； in, is the total number of samples, is the number of categories, For samples For category The true label, Predict samples for the model For category probability.

3. The short text topic identification method based on contrastive learning and LLMs according to claim 1 is characterized in that In step 9, the loss function for training the second multi-layer perceptron model based on the positive and negative sample pairs is ,in: ； in, is the total number of samples, and The samples and samples The vector representation of For samples and samples The square of the Euclidean distance between vector representations, Indicates whether the text pair is of the same category. If the sample and samples Belong to the same category, then , if the sample and samples Do not belong to the same category, then .

4. The short text topic identification method based on contrastive learning and LLMs according to claim 1 is characterized in that In step 8, selecting marginal texts and central texts in the second original text sample set from the clustering results includes: Calculating the confidence index and information entropy index of the classification prediction of the second multi-layer perceptron model; Calculating a comprehensive index based on the confidence index and the information entropy index; The marginal text is selected based on the comprehensive index.

5. The short text topic identification method based on contrastive learning and LLMs according to claim 4 is characterized in that: The calculation formula of the confidence index is: ； in, is the confidence index, is the category probability distribution output by the second multi-layer perceptron model, For samples The maximum probability of belonging to a cluster; The calculation formula of the information entropy index is: ； in, is the information entropy index, is the information entropy, is the number of sample clusters, Predict samples for the model For category probability; The calculation formula for calculating the comprehensive index based on the confidence index and the information entropy index is: ； in, is the comprehensive indicator, is the adjustment coefficient; The marginal text is selected based on the comprehensive index, including: If the If the preset indicator threshold is exceeded, the sample Marks the text for the edge.

6. The short text topic identification method based on contrastive learning and LLMs according to claim 1 is characterized in that In step 8, constructing a positive and negative sample pair set based on the edge text and the center text includes: Obtaining a cluster center vector in the clustering result; Combining the second enhanced vector corresponding to the central text with the cluster center vector of the same cluster as the central text as a positive text pair set; The second enhanced vector corresponding to the edge text and the cluster center vector of the same cluster are combined as a negative text pair set.

7. The short text topic identification method based on contrastive learning and LLMs according to claim 6 is characterized in that: The formula for obtaining the cluster center vector in the clustering result is: ； in, Cluster The cluster center vector, For samples The vector representation of For samples Belong to cluster The predicted probability of .

8. The short text topic identification method based on contrastive learning and LLMs according to claim 1 is characterized in that: In step 9, when training the second multi-layer perceptron model based on the positive and negative sample pairs, determining whether to stop iteration based on the cluster allocation change rate; The calculation formula of the cluster allocation change rate is: ； in, assigning rates of change to the clusters, is the total number of samples, For samples In the The clustering categories of the iterations, For samples In the The clustering categories of the iterations, is a binary indicator function; The determining whether to stop iteration based on the cluster allocation change rate includes: When the cluster allocation change rate is less than a preset change rate threshold, the iteration is stopped.

9. The short text topic identification method based on contrastive learning and LLMs according to claim 1, characterized in that: In step 9, the loss function for adjusting the first sentence embedding model based on the positive and negative sample pairs is ,in: ； in, is the positive text pair set, is a negative text pair set, is a positive sample pair The cosine similarity of is a negative sample pair The cosine similarity of is a hyperparameter.

10. The short text topic identification method based on contrastive learning and LLMs according to any one of claims 1 to 9, characterized in that: The method further comprises the following steps: Step 11: selecting marginal texts in the short text set to be identified from the recognition results as suspected abnormal texts; Step 12: Using a large language model to determine the correctness of the classification of the suspected abnormal text, if the classification is incorrect, the suspected abnormal text is marked as abnormal text, and the abnormal text is removed from the set of short texts to be identified, thereby obtaining a set of short texts to be extracted; Step 13: Based on the recognition result, the TF-IDF weighting method is used to extract the key words of each cluster in the short text set to be extracted.

Images