Short text topic extraction method fusing gaussian mixture model and large model fine-tuning
By integrating Gaussian mixture model and large language model, the topic distribution of short texts is optimized, which solves the problems of poor topic consistency and interpretability in short text scenarios and achieves efficient and reliable topic mining results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-05-20
- Publication Date
- 2026-08-04
AI Technical Summary
Traditional topic analysis methods are inefficient, costly, and susceptible to subjective bias in short text scenarios. Existing unsupervised topic models suffer from semantic ambiguity and poor internal consistency in short texts, making it difficult to meet the demands of high timeliness and large-scale text processing.
By integrating Gaussian mixture models and large language models, topic summaries are generated through data preprocessing, data augmentation, multivariate Gaussian mixture model construction, Dirichlet prior constraints, document-level contrastive learning, and large language models, thereby optimizing the semantic consistency and stability of topic distribution.
It significantly improves the accuracy and robustness of short text topic mining, ensures topic consistency and interpretability, and solves the problems of topic ambiguity and poor consistency in traditional methods.
Smart Images
Figure CN122220402B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of natural language processing and text mining technology, specifically involving a short text topic extraction method that integrates Gaussian mixture model and large model fine-tuning. Background Technology
[0002] With the rapid development of internet technology, the scale of text data is growing exponentially, with news platforms, social media, user comment systems, and other channels generating massive amounts of short text information daily. Against this backdrop, how to automatically mine potential themes and extract core semantic content from large-scale, noisy, and loosely structured short text collections has become a critical technical problem urgently needing to be solved in the field of natural language processing. Traditional topic analysis methods typically rely on domain experts for manual reading, summarization, and annotation. This approach is not only inefficient and costly but also susceptible to subjective cognitive biases, making it difficult to meet the current practical needs of high-timeliness and large-scale text processing.
[0003] To address this, researchers have proposed various unsupervised topic models, such as Latent Dirichlet Allocation (LDA), which aim to automatically infer latent topic structures and term distributions from text corpora. However, in short text scenarios, such models face significant challenges: on the one hand, short texts generally have characteristics such as sparse vocabulary, limited contextual information, and low word co-occurrence frequency; on the other hand, models based on the bag-of-words assumption, such as LDA, struggle to model semantic relationships, resulting in inferred topics often exhibiting semantic ambiguity, poor internal consistency, weak interpretability of topic terms, and poor stability, making it difficult to effectively support downstream applications.
[0004] In recent years, large language models have provided a new technical approach for topic modeling of short texts, thanks to their rich world knowledge and strong semantic generalization capabilities gained from pre-training on large-scale corpora. Large language models can generate highly abstract and semantically coherent topic descriptions based on natural language prompts and demonstrate cross-document semantic induction capabilities. However, relying solely on large language models for topic modeling still has certain limitations: the generated results are highly sensitive to the design of prompt words; the output content is limited by the model architecture and parameter size; and there is a lack of systematic evaluation metrics, making it difficult to achieve objective and quantifiable performance comparisons. Summary of the Invention
[0005] To address the issue of low topic consistency in existing topic models due to sparse semantic information and insufficient contextual information in short text data, this application proposes a short text topic extraction method that integrates Gaussian mixture models and large-scale model fine-tuning. This method first preprocesses and augments the short text documents, and then uses a pre-trained language model to obtain document-level and word-level embeddings. Based on this, a multivariate Gaussian mixture model is constructed to model the topic distribution of the documents. The rationality and semantic consistency of the topic distribution are improved by introducing a maximum mean difference constraint based on a Dirichlet prior and a document-level contrastive learning mechanism. Furthermore, a large-scale language model is used to semantically summarize the core words of the topics, generating topic summaries. Alignment constraints between the topic summary embeddings and Gaussian topic centers are constructed to optimize the consistency between the statistical topic space and the semantic representation space. Finally, the model is fine-tuned overall by combining multiple loss functions, thereby effectively improving the topic consistency and stability of short text topic mining.
[0006] To achieve the above objectives, this application employs the following technical solution:
[0007] This application presents a short text topic extraction method that integrates Gaussian mixture models and large model fine-tuning, specifically including the following steps:
[0008] Step 1: Perform data preprocessing and data augmentation on the short text document dataset to obtain short text documents. and short text documents Corresponding Enhanced Documents and for short text documents and enhanced documents Encode to obtain short text documents Document-level embedding representation, enhanced document Document-level embedding representation and word-level embedding representation;
[0009] Step 2, constructing a system containing... A multivariate Gaussian mixture model for each topic, used to analyze the short text documents obtained in step 1. Document-level embedding representation and enhanced document The document-level embedding representation is input into a multivariate Gaussian mixture model to calculate the probability of short text documents under each topic, and then... Function to get short text document Theme distribution and enhanced documents Theme distribution ;
[0010] Step 3: Introduce Dirichlet priors and apply maximum mean difference loss. For the short text document obtained in step 2 Theme distribution and enhanced documents Theme distribution Apply constraints while maintaining the original short text document. With Enhanced Documentation Construct document comparison learning loss between topic distributions To ensure semantic consistency;
[0011] Step 4: Project the word-level embeddings obtained in Step 1 onto the multivariate Gaussian mixture model constructed in Step 2 to calculate the topic-word distribution and extract the word-level embeddings for each topic. One core word; using a large language model, the core words are summarized into a topic summary, and the topic summary embedding vector is obtained by encoding through a pre-trained model;
[0012] Step 5: Construct an anchor-contrast loss by using the mean vector of the multivariate Gaussian mixture model and the corresponding topic summary embedding to form positive sample pairs, and using non-corresponding combinations as negative sample pairs. Combined with the maximum mean difference loss Document comparison learning loss Loss compared to anchor point Fine-tuning the multivariate Gaussian mixture model completes the topic extraction of the paragraph text.
[0013] A further improvement of this application is that step 1 specifically includes the following steps:
[0014] Step 1.1: Perform data preprocessing operations on the short text document dataset, including removing stop words, eliminating non-letter symbols, and performing lemmatization; to obtain the preprocessed short text documents. ;
[0015] Step 1.2: Process the preprocessed short text documents based on the WordNet lexicon. Performing data augmentation involves using WordNet to replace words with synonyms and generate short text documents. Corresponding Enhanced Documents ;
[0016] Step 1.3: Use a pre-trained model to process short text documents. and enhanced documents Encoding is performed to obtain document-level embedding representations and word-level embedding representations, where the first... A short text document Document-level embedding is represented as Enhance documents Document-level embedding is represented as The first in the vocabulary list The word is recorded as , No. Word The word-level embedding representation is denoted as .
[0017] A further improvement of this application is that step 2 specifically includes the following steps:
[0018] Step 2.1, take the first result obtained in step 1.3... A short text document Document-level embedding is represented as Enhanced Documents Document-level embedding is represented as Inputting the data into a multivariate Gaussian mixture model, the model calculates which short text document belongs to the first... The probability of the i-th topic, where the i-th topic is... The probability density function of a topic is defined as:
[0019]
[0020]
[0021] in, For the first The mean vector of each topic, For the first The covariance matrix of a Gaussian model Indicates the dimension of the embedding space. For transpose, Number of topics;
[0022] Step 2.2: Based on the document topic probability under each topic Representing document-topic distribution, specifically, through Function to get short text document Theme distribution and enhanced documents Theme distribution :
[0023]
[0024]
[0025] in, Temperature parameters normalized for document-topic distribution, used for control The smoothness of the function.
[0026] A further improvement of this application is that step 3 specifically includes the following steps:
[0027] Step 3.1: In the calculated short text document Theme distribution and enhanced documents Theme distribution Introducing the Dirichlet prior, based on the maximum mean difference loss. For short text documents Theme distribution and enhanced documents Theme distribution Apply constraints, posterior With Dirichlet's a priori The maximum mean difference between them is defined as:
[0028]
[0029] in, To obtain from the posterior distribution The resulting batch of documents and their topic distributions were obtained through encoding. This indicates the document-topic distribution. Mapped to one-dimensional real space The reproducing kernel Hilbert space (RKHS) of real-valued functions. Indicates will A positive definite kernel mapped to a high-dimensional feature space. From short text document Theme distribution and enhanced documents Theme distribution It is pieced together. Indicates batch, from Dirichlet a priori A batch of samples were distributed as follows Maximum mean difference loss The estimate is:
[0030]
[0031] in, Indicates batch size, This represents an index variable, with a value range of 1000. ; Indicates that it comes from the posterior distribution The first obtained from Document-Topic Distribution Sample and the first Document-topic distribution sample, Indicates a priori from Dirichlet. The first sample obtained from the middle Document-Topic Distribution Sample and the first Document-topic distribution sample;
[0032] Step 3.2: In actual training, due to computational resource limitations, it is impossible to process all documents at once. Instead, a batch processing approach is used, calculating only a portion of the documents at a time. Batch documents to obtain short text documents Single-batch topic distribution and enhanced documents Single-batch topic distribution ;
[0033] Step 3.3: Based on the calculated short text document Single-batch topic distribution and enhanced documents Single-batch topic distribution Build batch size is Document comparison learning loss at time ;
[0034]
[0035] in, Positive sample pairs are defined as the topic distributions of the same short text document and its corresponding enhanced document. Negative sample pairs are defined as the topic distributions of all other document types in the same batch. This represents a similarity measure between vectors in short text documents. Single-batch topic distribution and enhanced documents Single-batch topic distribution When normalization has been performed, the similarity is equivalent to cosine similarity; Temperature parameters for document comparison learning are used to adjust the smoothness of the distribution;
[0036] Step 3.4: Loss based on maximum mean difference Learning loss compared with document The overall loss function of the multivariate Gaussian mixture model Defined as:
[0037]
[0038] in, This is a weighting factor used to balance the contribution of the maximum mean difference loss.
[0039] A further improvement of this application is that step 4 specifically includes the following steps:
[0040] Step 4.1: Denote the word-level embedding representation obtained in Step 1.3 as The word-level embedding representation is computed in the multivariate Gaussian mixture model with fixed parameters at the 1st digit. Probability under each topic:
[0041]
[0042] in, For the first The mean vector of each topic, For the first The covariance matrix of a multivariate Gaussian mixture model Indicates the dimension of the embedding space;
[0043] Based on each word in the Probability under each topic Calculate the first Theme-word distribution of each topic :
[0044]
[0045] in, This indicates the total number of words in the vocabulary list;
[0046] Step 4.2: For each topic, analyze the topic-word distribution. Select the one with the highest probability Using a large language model, semantic summaries are generated from core words to produce topic summaries. These topic summaries are then encoded using a pre-trained model to obtain corresponding topic summary embedding vectors. .
[0047] A further improvement of this application is that step 5 specifically includes the following steps:
[0048] Step 5.1, the first step based on the multivariate Gaussian mixture model Mean vector of each topic With the obtained topic summary embedding vector Construct a contrastive learning objective, where the first... Mean vector of each topic Its corresponding topic summary embedding Constitute a positive sample pair, the first Mean vector of each topic With the subject summary embedded Other topic summary embedding vectors For negative sample pairs, Construct anchor points to compare losses :
[0049]
[0050] in, This represents a similarity measure between vectors. When vectors are normalized, the similarity measure is equivalent to cosine similarity. The temperature parameter for anchor point comparison loss is used to adjust the smoothness of the distribution;
[0051] Step 5.2: Overall loss function based on multivariate Gaussian mixture model Loss compared with anchor point Construct the overall fine-tuning loss function for the multivariate Gaussian mixture model. :
[0052] .
[0053] The beneficial effects of this application are:
[0054] This application introduces a large language model to generate a natural language summary of the topic from representative core words, which greatly improves the interpretability of the topic and solves the problem of vague and difficult-to-understand topic expression in traditional topic models.
[0055] This application achieves precise matching between documents and topics by constructing a semantic alignment mechanism between Gaussian topic centers and topic summary vectors. This ensures semantic coherence and consistency within each topic and clear and distinguishable boundaries between topics, effectively avoiding topic confusion. It significantly enhances the robustness and operational reliability of the entire topic mining system and guarantees the accuracy and credibility of topic mining results.
[0056] This application specifically overcomes the problem of low topic consistency in traditional topic models in short text scenarios due to insufficient contextual information. Through multi-loss joint optimization and semantic alignment design, it significantly improves the accuracy, robustness, and interpretability of short text topic modeling, providing an efficient and reliable technical solution for short text topic mining, and has strong practical application value. Attached Figure Description
[0057] Figure 1 This is a model structure diagram of this application.
[0058] Figure 2 This is a flowchart of this application.
[0059] Figure 3 This is a line graph comparing the different comparison models in this application across four indicators. Detailed Implementation
[0060] The specific embodiments of this application are described below with reference to the accompanying drawings. To make the technical solution of this application clearer and more complete, several specific implementation details will be described below. However, it should be understood that these specific implementation details are for illustrative purposes only and are not intended to limit this application; that is, in some embodiments of this application, the details may be omitted or replaced by other equivalent methods. In addition, for the sake of clarity of illustration, well-known structures and conventional components in the relevant technical field have been simplified in the accompanying drawings and are shown only in a schematic manner.
[0061] like Figure 1As shown, this application presents a short text topic extraction method that integrates Gaussian mixture models and large model fine-tuning, specifically including the following steps:
[0062] Step 1: Perform data preprocessing and data augmentation on the short text document dataset to obtain short text documents. and short text documents Corresponding Enhanced Documents and for short text documents and enhanced documents Encode to obtain short text documents Document-level embedding representation, enhanced document Document-level and word-level embedding representations. Specifically, the steps include:
[0063] Step 1.1: Perform data preprocessing operations on the short text document dataset, including removing stop words, eliminating non-letter symbols, and performing lemmatization; to obtain the preprocessed short text documents. ;
[0064] Step 1.2: Process the preprocessed short text documents based on the WordNet lexicon. Performing data augmentation involves using WordNet to replace words with synonyms and generate short text documents. Corresponding Enhanced Documents ;
[0065] Step 1.3: Use a pre-trained model, i.e., for short text documents. and enhanced documents Encoding is performed to obtain document-level embedding representations and word-level embedding representations, where the first... A short text document Document-level embedding is represented as Enhance documents Document-level embedding is represented as The first in the vocabulary list The word is recorded as , No. Word The word-level embedding representation is denoted as The pre-trained model in this application is a Transformer-based pre-trained language model that uses the original weights and does not participate in fine-tuning.
[0066] Step 2, constructing a system containing... A multivariate Gaussian mixture model for each topic, used to analyze the short text documents obtained in step 1. Document-level embedding representation and enhanced document The document-level embedding representation is input into a multivariate Gaussian mixture model to calculate the probability of short text documents under each topic, and then... Function to get short text document Theme distribution and enhanced documents Theme distribution Specifically, it includes the following steps:
[0067] Step 2.1, take the first result obtained in step 1.3... A short text document Document-level embedding is represented as Enhanced Documents Document-level embedding is represented as Inputting the data into a multivariate Gaussian mixture model, the model calculates which short text document belongs to the first... The probability of the i-th topic, where the i-th topic is... The probability density function of a topic is defined as:
[0068]
[0069]
[0070] in, For the first The mean vector of each topic, For the first The covariance matrix of a Gaussian model Indicates the dimension of the embedding space. For transpose, Number of topics;
[0071] Step 2.2: Based on the document topic probability under each topic Representing document-topic distribution, specifically, through Function to get short text document Theme distribution and enhanced documents Theme distribution :
[0072]
[0073]
[0074] in, Temperature parameters normalized for document-topic distribution, used for control The smoothness of the function.
[0075] Step 3: Introduce Dirichlet priors and apply maximum mean difference loss. For the short text document obtained in step 2 Theme distribution and enhanced documents Theme distribution Apply constraints while maintaining the original short text document. With Enhanced Documentation Construct document comparison learning loss between topic distributions This ensures semantic consistency. Specifically, it includes the following steps:
[0076] Step 3.1: In the calculated short text document Theme distribution and enhanced documents Theme distribution Introducing the Dirichlet prior, based on the maximum mean difference loss. For short text documents Theme distribution and enhanced documents Theme distribution Apply constraints, posterior With Dirichlet's a priori The maximum mean difference between them is defined as:
[0077]
[0078] in, To obtain from the posterior distribution The resulting batch of documents and their topic distributions were obtained through encoding. This indicates the document-topic distribution. Mapped to one-dimensional real space The reproducing kernel Hilbert space (RKHS) of real-valued functions. Indicates will A positive definite kernel mapped to a high-dimensional feature space. From short text document Theme distribution and enhanced documents Theme distribution It is pieced together. Indicates batch, from Dirichlet a priori A batch of samples were distributed as follows Maximum mean difference loss The estimate is:
[0079]
[0080] in, Indicates batch size, This represents an index variable, with a value range of 1000. ; Indicates that it comes from the posterior distribution The first obtained from Document-Topic Distribution Sample and the first Document-topic distribution sample, Indicates a priori from Dirichlet. The first sample obtained from the middle Document-Topic Distribution Sample and the first Document-topic distribution sample;
[0081] Step 3.2: In actual training, due to computational resource limitations, it is impossible to process all documents at once. Instead, a batch processing approach is used, calculating only a portion of the documents at a time. Batch documents to obtain short text documents Single-batch topic distribution and enhanced documents Single-batch topic distribution ;
[0082] Step 3.3: Based on the calculated short text document Single-batch topic distribution and enhanced documents Single-batch topic distribution Build batch size is Document comparison learning loss at time ;
[0083]
[0084] in, Positive sample pairs are defined as the topic distributions of the same short text document and its corresponding enhanced document. Negative sample pairs are defined as the topic distributions of all other document types in the same batch. This represents a similarity measure between vectors in short text documents. Single-batch topic distribution and enhanced documents Single-batch topic distribution When normalization has been performed, the similarity is equivalent to cosine similarity; Temperature parameters are used for document comparison learning to adjust the smoothness of the distribution.
[0085] Step 3.4: Loss based on maximum mean difference Learning loss compared with document The overall loss function of the multivariate Gaussian mixture model Defined as:
[0086]
[0087] in, This is a weighting factor used to balance the contribution of the maximum mean difference loss;
[0088] Step 4: Project the word-level embeddings obtained in Step 1 onto the multivariate Gaussian mixture model constructed in Step 2 to calculate the topic-word distribution and extract the word-level embeddings for each topic. The process involves identifying core words, summarizing them into topic summaries using a large language model, and encoding these topic summaries into embedding vectors using a pre-trained model. The specific steps include:
[0089] Step 4.1: Denote the word-level embedding representation obtained in Step 1.3 as The word-level embedding representation is computed in the multivariate Gaussian mixture model with fixed parameters at the 1st digit. Probability under each topic:
[0090]
[0091] in, For the first The mean vector of each topic, For the first The covariance matrix of a multivariate Gaussian mixture model Indicates the dimension of the embedding space;
[0092] Based on each word in the Probability under each topic Calculate the first Theme-word distribution of each topic :
[0093]
[0094] in, This indicates the total number of words in the vocabulary list;
[0095] Step 4.2: For each topic, analyze the topic-word distribution. Select the one with the highest probability Using a large language model, semantic summaries are generated from core words to produce topic summaries. These topic summaries are then encoded using a pre-trained model to obtain corresponding topic summary embedding vectors. The pre-trained model here is a Transformer-based pre-trained language model, which does not require fine-tuning. The purpose is to ensure the quality of short text documents. Mean vector The input and output should reside in the same semantic space as the topic summary. The prompt is: Given a set of core words representing a potential topic, generate a concise, high-level summary that captures the common meaning of these words. The summary must contain only 3 to 5 words, must not directly repeat the original core words, should use abstract terminology as much as possible, and should only return a topic summary without any explanation. The input is the core words, and the output is the summary.
[0096] Step 5: Construct an anchor-contrast loss by using the mean vector of the multivariate Gaussian mixture model and the corresponding topic summary embedding to form positive sample pairs, and using non-corresponding combinations as negative sample pairs. Combined with the maximum mean difference loss Document comparison learning loss Loss compared to anchor point Fine-tuning the multivariate Gaussian mixture model involves incorporating the three loss functions mentioned above into the model training process and iteratively updating the model parameters based on the joint loss using the backpropagation algorithm. Specifically, this includes the following steps:
[0097] Step 5.1, the first step based on the multivariate Gaussian mixture model Mean vector of each topic With the obtained topic summary embedding vector Construct a contrastive learning objective, where the first... Mean vector of each topic Its corresponding topic summary embedding Constitute a positive sample pair, the first Mean vector of each topic With the subject summary embedded Other topic summary embedding vectors For negative sample pairs, Construct anchor points to compare losses :
[0098]
[0099] in, This represents a similarity measure between vectors. When vectors are normalized, the similarity measure is equivalent to cosine similarity. The temperature parameter for anchor point comparison loss is used to adjust the smoothness of the distribution;
[0100] Step 5.2: Overall loss function based on multivariate Gaussian mixture model Loss compared with anchor point Construct the overall fine-tuning loss function for the multivariate Gaussian mixture model. :
[0101] .
[0102] To verify this application, experiments were conducted on a short text document dataset. The relevant experimental data on the short text document dataset are shown in Table 1:
[0103] Table 1 shows the numerical values of topic consistency and diversity for this application tested on the HuffPostNews dataset.
[0104]
[0105] As shown in Table 1, the topic consistency and diversity scores of this application tested on the HuffPostNews dataset are as follows: CP = 0.524, CA = 0.260, NPMI = 0.093, and UT = 0.936. All scores are higher than those in the comparative experiments, with the highest scores in the comparative experiments being CP = 0.296, CA = 0.202, NPMI = 0.042, and UT = 0.800. The comparative models used in this experiment are Comparative Example 1, Comparative Example 2, and Comparative Example 3. The specific model sources are as follows:
[0106] Comparative Example 1 uses the LDA method; Comparative Example 2 uses the CAST method; Comparative Example 3 uses the LLM-TE method.
[0107] Experimental results show that the proposed method significantly outperforms existing methods in terms of topic consistency and diversity metrics (CP, CA, NPMI, UT) for short text corpora, verifying the advantage of the proposed method in maintaining semantic consistency on short text datasets.
[0108] This embodiment is only used to illustrate a feasible implementation of the technical solution of the present invention and does not constitute a limitation on the scope of protection of the claims. Without departing from the technical concept defined by the claims of the present invention, reasonable derivations made by those skilled in the art based on the content disclosed in the specification, such as adaptive adjustments, equivalent substitutions, or scenario expansions of model parameters, network structures, or application environments, should be considered as falling within the scope of the original inventive contribution of this application and are protected by patent rights in accordance with the law.
Claims
1. A method for extracting short text topics by integrating Gaussian mixture models and large model fine-tuning, characterized in that: The short text topic extraction method specifically includes the following steps: Step 1: Perform data preprocessing and data augmentation on the short text document dataset to obtain short text documents. and short text documents Corresponding Enhanced Documents and for short text documents and enhanced documents Encode to obtain short text documents Document-level embedding representation, enhanced document Document-level embedding representation and word-level embedding representation; Step 2, constructing a system containing... A multivariate Gaussian mixture model for each topic, used to analyze the short text documents obtained in step 1. Document-level embedding representation and enhanced document The document-level embedding representation is input into a multivariate Gaussian mixture model to calculate the probability of short text documents under each topic, and then... Function to get short text document Theme distribution and enhanced documents Theme distribution ; Step 3: Introduce Dirichlet priors and apply maximum mean difference loss. For the short text document obtained in step 2 Theme distribution and enhanced documents Theme distribution Apply constraints while maintaining the original short text document. With Enhanced Documentation Construct document comparison learning loss between topic distributions To ensure semantic consistency; Step 4: Project the word-level embeddings obtained in Step 1 onto the multivariate Gaussian mixture model constructed in Step 2 to calculate the topic-word distribution and extract the word-level embeddings for each topic. One core word; using a large language model, the core words are summarized into a topic summary, and the topic summary embedding vector is obtained by encoding through a pre-trained model; Step 5: Construct an anchor-contrast loss by using the mean vector of the multivariate Gaussian mixture model and the corresponding topic summary embedding to form positive sample pairs, and using non-corresponding combinations as negative sample pairs. Combined with the maximum mean difference loss Document comparison learning loss Loss compared to anchor point Fine-tuning of a multivariate Gaussian mixture model to extract topics from short texts, including anchor point contrast loss. The construction method is as follows: Based on the multivariate Gaussian mixture model, the first Mean vector of each topic With the obtained topic summary embedding vector Construct a contrastive learning objective, where the first... Mean vector of each topic Its corresponding topic summary embedding Constitute a positive sample pair, the first Mean vector of each topic With the subject summary embedded Other topic summary embedding vectors For negative sample pairs, Construct anchor points to compare losses : in, This represents a similarity measure between vectors. When vectors are normalized, the similarity measure is equivalent to cosine similarity. The temperature parameter is used to compare the loss at the anchor point and adjust the smoothness of the distribution.
2. The short text topic extraction method integrating Gaussian mixture model and large model fine-tuning as described in claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1.1: Perform data preprocessing on the short text document dataset to obtain preprocessed short text documents. ; Step 1.2: Process the preprocessed short text documents based on the WordNet lexicon. Perform data augmentation operations to generate short text documents. Corresponding Enhanced Documents ; Step 1.3: Use a pre-trained model to process short text documents. and enhanced documents Encoding is performed to obtain document-level embedding representations and word-level embedding representations, where the first... A short text document Document-level embedding is represented as Enhance documents Document-level embedding is represented as The first in the vocabulary list The word is recorded as , No. Word The word-level embedding representation is denoted as .
3. The short text topic extraction method integrating Gaussian mixture model and large model fine-tuning as described in claim 2, characterized in that: Step 2 specifically includes the following steps: Step 2.1, take the first result obtained in step 1.3... A short text document Document-level embedding is represented as Enhanced Documents Document-level embedding is represented as Inputting the data into a multivariate Gaussian mixture model, the model calculates which short text document belongs to the first... The probability of the i-th topic, where the i-th topic is... The probability density function of a topic is defined as: in, For the first The mean vector of each topic, For the first The covariance matrix of a Gaussian model Indicates the dimension of the embedding space. For transpose, Number of topics; Step 2.2, through Function to get short text document Theme distribution and enhanced documents Theme distribution : in, Temperature parameters normalized for document-topic distribution, used for control The smoothness of the function.
4. The short text topic extraction method integrating Gaussian mixture model and large model fine-tuning as described in claim 3, characterized in that: Step 3 specifically includes the following steps: Step 3.1: In the calculated short text document Theme distribution and enhanced documents Theme distribution Introducing the Dirichlet prior, based on the maximum mean difference loss. For short text documents Theme distribution and enhanced documents Theme distribution Apply constraints, posterior With Dirichlet's a priori The maximum mean difference between them is defined as: in, To obtain from the posterior distribution The encoded document-topic distribution This indicates the document-topic distribution. Mapped to one-dimensional real space The reproducing kernel Hilbert space of real-valued functions, Indicates will A positive definite kernel mapped to a high-dimensional feature space. From short text document Theme distribution and enhanced documents Theme distribution It is pieced together. Indicates batch, from Dirichlet a priori A batch of samples were distributed as follows Maximum mean difference loss The estimate is: in, Indicates batch size, This represents an index variable, with a value range of 1000. ; Indicates that it comes from the posterior distribution The first obtained from Document-Topic Distribution Sample and the first Document-topic distribution sample, Indicates a priori from Dirichlet. The first sample obtained from the middle Document-Topic Distribution Sample and the first Document-topic distribution sample; Step 3.2: Calculate only once. Batch documents to obtain short text documents Single-batch topic distribution and enhanced documents Single-batch topic distribution ; Step 3.3: Based on the calculated short text document Single-batch topic distribution and enhanced documents Single-batch topic distribution Build batch size is Document comparison learning loss at time ; in, Positive sample pairs are defined as the topic distributions of the same short text document and its corresponding enhanced document. Negative sample pairs are defined as the topic distributions of all other document types in the same batch. This represents a measure of similarity between vectors. Temperature parameters for document comparison learning are used to adjust the smoothness of the distribution; Step 3.4: Loss based on maximum mean difference Learning loss compared with document The overall loss function of the multivariate Gaussian mixture model Defined as: in, This is a weighting factor used to balance the contribution of the maximum mean difference loss.
5. The short text topic extraction method based on the fusion of Gaussian mixture model and large model fine-tuning as described in claim 4, characterized in that: Step 4 specifically includes the following steps: Step 4.1: Denote the word-level embedding representation obtained in Step 1.3 as The input is fed into a multivariate Gaussian mixture model to compute the word-level embedding representation at the th... Probability under each topic: in, For the first The mean vector of each topic, For the first The covariance matrix of a multivariate Gaussian mixture model Indicates the dimension of the embedding space; Based on each word in the Probability under each topic Calculate the first Theme-word distribution of each topic : in, This indicates the total number of words in the vocabulary list; Step 4.2: For each topic, analyze the topic-word distribution. Select the one with the highest probability Using a large language model, semantic summaries are generated from core words to produce topic summaries. These topic summaries are then encoded using a pre-trained model to obtain corresponding topic summary embedding vectors. .
6. The short text topic extraction method based on the fusion of Gaussian mixture model and large model fine-tuning as described in claim 5, characterized in that: In step 5, the overall loss function is based on the multivariate Gaussian mixture model. Loss compared with anchor point Construct the overall fine-tuning loss function for the multivariate Gaussian mixture model. : 。