Short text clustering method based on adaptive optimal transmission and three-level robust representation

Abstract

The invention provides a short text clustering method based on adaptive optimal transmission and three-level robust representation, which comprises the following steps of: 1, coding an input original short text based on a pre-training language model in a server carrying a GPU (Graphics Processing Unit) to obtain a semantic representation of the original short text, and performing multi-type data enhancement processing on the semantic representation to obtain a semantic representation of the original short text; obtaining an enhanced sample representation; 2, performing pseudo label distribution on enhanced sample representation based on an optimal transmission algorithm; 3, learning and calculating a corresponding loss function based on the pseudo label and three-level robust representation, and updating model parameters of the loss function in a CPU (Central Processing Unit) of a server; 4, iteratively executing the steps 1-3, and outputting a target clustering result when a clustering result meets a convergence condition; the problems of data sparseness and class imbalance in short text clustering can be effectively relieved, and the stability and precision of a clustering result are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent analysis and machine learning technology for short text data in natural language processing, and particularly to a short text clustering method based on adaptive optimal transmission and three-level robust representation. Background Technology

[0002] Currently, with the widespread adoption of the internet and mobile devices, platforms such as social media, e-commerce, news, and online forums generate massive amounts of user-generated content daily, a significant portion of which exists in the form of short texts, such as microblog posts, user comments, product reviews, and search keywords. Short texts are typically concise, information-dense, and thematically focused, but due to a lack of rich context and sufficient vocabulary, their semantic information exhibits sparsity and incompleteness, posing a significant challenge to feature modeling and semantic understanding in natural language processing. Against the backdrop of the continuous development of big data and artificial intelligence technologies, how to efficiently and accurately cluster short texts automatically without manual annotation has become a crucial fundamental issue in text mining, information retrieval, recommendation systems, and public opinion analysis. Unsupervised clustering methods, capable of automatically classifying samples based on similarity, have significant application value in short text processing tasks. However, compared to long texts, short texts lack contextual information, and their feature representation is sparse and lacks discriminative power, limiting the performance of traditional clustering methods in such tasks.

[0003] However, existing short text clustering techniques mainly include methods based on traditional feature engineering and methods based on deep representation learning. Traditional methods typically construct sparse feature vectors using models such as bag-of-words and TF-IDF, and then combine them with algorithms such as K-means and hierarchical clustering for clustering. However, in short text scenarios, due to the high feature dimensionality and insufficient semantic discriminative power, these methods often fail to achieve ideal clustering results. In recent years, deep learning methods have been introduced into the field of short text processing. They automatically extract semantic embedding representations through models such as convolutional neural networks, recurrent neural networks, and Transformers, and optimize the feature space structure with techniques such as contrastive learning, thereby alleviating the problem of insufficient feature representation capabilities of traditional methods to some extent. However, these methods still rely on data augmentation to generate positive and negative sample pairs, and often use single augmentation methods such as synonym substitution and random masking, which can easily lead to insufficient sample diversity or decreased semantic consistency, thus affecting the model's discriminative ability. Furthermore, in unsupervised or semi-supervised clustering tasks, existing methods often use iterative generation of pseudo-labels to improve model performance. However, since the initial clustering results inevitably contain noisy labels, these erroneous labels accumulate in subsequent training iterations, causing the cluster centers to shift or even multiple clusters to collapse into a single category, resulting in cluster degradation. At the same time, many methods optimize only the relative distance between sample pairs, lacking global constraints on the class-level structure, resulting in insufficient intra-class aggregation and insufficient inter-class separation, further exacerbating the risk of cluster degradation.

[0004] Therefore, in order to overcome the above-mentioned technical problems, the present invention provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. Summary of the Invention

[0005] This invention provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. It encodes short texts using a pre-trained language model and enhances sample diversity through explicit and virtual augmentation strategies. The encoded results are then input into a pseudo-label generation module, which dynamically solves the transmission matrix based on a discrete optimal transmission framework combined with entropy regularization and a distribution variable penalty function to generate high-quality pseudo-labels. Next, guided by the pseudo-labels, three mechanisms—prototype learning, class comparison learning, and instance comparison learning—are executed to optimize feature representations at three levels: intra-class aggregation, inter-class differentiation, and sample discrimination, respectively. Finally, iterative training through a warm-up phase and a joint optimization phase achieves closed-loop optimization of feature extraction, pseudo-label generation, and robust representation learning. Training terminates when the cluster assignment change between two consecutive iterations falls below a preset threshold or reaches the maximum number of iterations. This invention effectively alleviates the problems of data sparsity and class imbalance in short text clustering, improving the stability and accuracy of clustering results.

[0006] A short text clustering method based on adaptive optimal transmission and three-level robust representation includes: Step 1: On a server equipped with a GPU, the input original short text is encoded based on a pre-trained language model to obtain the semantic representation of the original short text, and the semantic representation is subjected to multi-type data augmentation processing to obtain augmented sample representations; Step 2: Assign pseudo-labels to the augmented sample representations based on the optimal transmission algorithm; Step 3: Calculate the corresponding loss function based on the pseudo-labels and the three-level robust representation learning, and update the model parameters in the CPU of the server using the loss function; Step 4: Iterate through steps 1-3, and output the target clustering result when the clustering result meets the convergence condition.

[0007] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation performs multi-type data augmentation processing on semantic representation to obtain enhanced sample representation, including: Read the semantic representation of the original short text and determine the embedding vector of the original short text; The original short text is replaced with keywords to generate explicit augmented samples. At the same time, the embedded vector is randomly erased to generate virtual augmented samples.

[0008] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation includes: The weight values ​​of each word in the preset vocabulary set are calculated based on the pre-trained BERT model. The preset vocabulary set is sorted according to the weight values ​​of each word. The top preset proportion of words are selected as target words to replace keywords in the original short text, generating explicit augmented samples. When performing random erasure on the embedded vector, the second preset proportion of words in the embedded vector are masked.

[0009] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation includes, in step 2, assigning pseudo-labels to the enhanced sample representations based on the optimal transmission algorithm, including: The original text is mapped into feature representations based on encoding networks; The feature representation is processed according to the clustering network, and the clustering probability of the sample is predicted based on the processing result. A cost matrix is ​​constructed based on the clustering assignment probability of the samples, and the migration cost from samples to categories is quantified based on the cost matrix. Construct an optimization objective, and determine the association matrix between samples and categories based on the optimization objective, wherein the optimization objective is constrained by an entropy regularization term and a distribution variable penalty function; The correlation matrix satisfying the optimization objective is dynamically calculated according to a preset iterative algorithm; Based on the association matrix, the category with the highest association strength for each sample is output, and the category with the highest association strength is used as the pseudo-label for assignment.

[0010] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation includes step 3, which involves learning and calculating the corresponding loss function based on the pseudo-labels and the three-level robust representation, and updating the model parameters in the server's CPU using the loss function, including: The three-level robust representation learning includes: execution prototype learning, class comparison learning, and instance comparison learning; Specifically, the prototype learning maintains a learnable prototype vector for each category and achieves intra-class aggregation through an improved cross-entropy loss; the class contrastive learning constrains different augmented views of the same original text to maintain consistency in the prediction distribution; and the instance contrastive learning brings the projected representations of positive sample pairs closer together and pushes the projected representations of negative sample pairs further apart in the projection space. The cumulative sum is calculated based on the loss functions of prototype learning, class comparison learning, and instance comparison learning to obtain the total loss function.

[0011] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation uses the following loss function for prototype learning: ; in, Represents the prototype vector The loss function; Represents the prototype vector; Indicates batch size; This represents the sample's serial number value. Represents the logarithmic function with base e; Indicates the first Features of each sample; Indicates the first Pseudo-labels for individual samples The prototype vector; Indicates the first The pseudo-labels of the samples; Indicates the total number of categories; The index value representing the category; Indicates the first The prototype vectors corresponding to each category; This indicates temperature hyperparameters; Represents the similarity function; This represents an exponential function.

[0012] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation uses the following loss function for class contrastive learning: ; in, This represents the loss function used to predict the probability distribution of classes in contrastive learning. This represents the predicted category probability distribution value; Indicates the first One sample in virtual augmented view The predicted category probability distribution values ​​are as follows; Indicates the first One sample is displayed in the enhanced view. The predicted category probability distribution values.

[0013] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation uses the following loss function for instance contrastive learning: positive sample pairs are defined as augmented samples from the same original text, and negative sample pairs are defined as augmented samples from different original texts; for all augmented samples in a batch, their projected representations are used to calculate the loss, and the loss function is as follows: ; in, The loss function for instance-based comparative learning; Indicates comparative loss; Indicates the first Contrast loss between virtual enhancement and display enhancement for each sample; Indicates the first Contrast loss between visual enhancement and virtual enhancement for each sample.

[0014] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation calculates the cumulative sum of the loss functions of prototype learning, class contrastive learning, and instance contrastive learning to obtain the total loss function, including: ; in, , , For weight hyperparameters.

[0015] Preferably, a short text clustering method based on adaptive optimal transmission and three-level robust representation includes step 4, iteratively executing steps 1-3, and outputting the target clustering result when the clustering result meets the convergence condition, including: S41: Use K-means to cluster the initial feature representations to generate initial pseudo-labels; S42: Under pseudo-label supervision, iteratively optimize the parameters of Sentence-BERT, prototype tensor, clustering head and comparison head, and dynamically update pseudo-labels in each iteration; S43: Training terminates when the change in clustering assignment results between two consecutive iterations is lower than a preset threshold or the maximum number of iterations is reached. S44: The clustering results are determined by the category label corresponding to the highest probability in each row of the prediction matrix.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention encodes short texts using a pre-trained language model and enhances sample diversity through explicit and virtual augmentation strategies. The encoded results are then input into a pseudo-label generation module, which dynamically solves the transfer matrix based on a discrete optimal transfer framework combined with entropy regularization and a distribution variable penalty function, thereby generating high-quality pseudo-labels. Next, guided by the pseudo-labels, three mechanisms—prototype learning, class comparison learning, and instance comparison learning—are implemented to optimize feature representations at three levels: intra-class aggregation, inter-class differentiation, and sample discrimination, respectively. Finally, iterative training through a warm-up phase and a joint optimization phase achieves closed-loop optimization of feature extraction, pseudo-label generation, and robust representation learning. Training terminates when the cluster assignment change between two consecutive iterations falls below a preset threshold or reaches the maximum number of iterations. This invention effectively alleviates the problems of data sparsity and class imbalance in short text clustering, improving the stability and accuracy of clustering results.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a short text clustering method based on adaptive optimal transmission and three-level robust representation in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall framework of a short text clustering method based on adaptive optimal transmission and three-level robust representation. Figure 3 This is a schematic diagram of the feature extraction structure of a short text clustering method based on adaptive optimal transmission and three-level robust representation. Figure 4 A flowchart of pseudo-label generation for a short text clustering method based on adaptive optimal transmission and three-level robust representation; Figure 5 This is a schematic diagram of the three-level robust representation learning structure of a short text clustering method based on adaptive optimal transmission and three-level robust representation. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Example 1: This example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation, such as... Figure 1 As shown, it includes: Step 1: On a server equipped with a GPU, the input original short text is encoded based on a pre-trained language model to obtain the semantic representation of the original short text, and the semantic representation is subjected to multi-type data augmentation processing to obtain augmented sample representations; Step 2: Assign pseudo-labels to the augmented sample representations based on the optimal transmission algorithm; Step 3: Calculate the corresponding loss function based on the pseudo-labels and the three-level robust representation learning, and update the model parameters in the CPU of the server using the loss function; Step 4: Iterate through steps 1-3, and output the target clustering result when the clustering result meets the convergence condition.

[0022] In this embodiment, updating the model parameters means that the model includes the Sentence-BERT model (i.e., the pre-trained language model), the prototype tensor, the cluster head, and the comparison head. The parameters to be updated are the Sentence-BERT network weights determined according to the loss function, the vector coordinates of each prototype in the prototype tensor, and the network weights in the cluster head and the comparison head.

[0023] In this embodiment, the original short text includes, for example, Weibo content, user comments, product reviews, search keywords, etc.

[0024] In this embodiment, three mechanisms—prototype learning, class contrast learning, and instance contrast learning—are executed under the guidance of pseudo-labels to optimize feature representation from three levels: intra-class aggregation, inter-class differentiation, and sample discrimination, respectively. Specifically, the three-level robust representation calculation involves executing prototype learning, class contrast learning, and instance contrast learning under the guidance of pseudo-labels; calculating the total training loss; prototype learning maintains a learnable prototype vector for each category and achieves intra-class aggregation through an improved cross-entropy loss; class contrast learning constrains the prediction distribution of different augmented views of the same text to remain consistent; instance contrast learning brings positive sample pairs closer together and pushes negative sample pairs further apart in the projection space; and the three losses are weighted and summed according to preset weights to obtain the total loss function.

[0025] In this embodiment, an optimal transmission cost matrix is ​​constructed based on the clustering probability distribution of the batch texts. The cost matrix, entropy regularization term, and distribution variable penalty function together constitute the optimization objective. The optimization objective is solved by dynamically updating the class distribution variables to obtain the transmission matrix, and pseudo-labels are assigned according to the transmission matrix.

[0026] In this embodiment, semantic representation refers to a string of "digital ID cards" or "feature codes" generated by the computer after understanding the original short text.

[0027] In this embodiment, feature extraction and data augmentation refer to using a BERT pre-trained model to replace keywords to generate explicit augmented samples; performing explicit augmentation and virtual augmentation; using a pre-trained Sentence-BERT model (i.e., a pre-trained language model) to encode short text input to obtain feature space embedding vectors of the original text and the explicitly augmented text; and using a random erasure strategy on the original text embedding vector to generate virtual augmented samples.

[0028] In this embodiment, the parameters of Sentence-BERT, prototype tensor, cluster head and comparison head are iteratively optimized under pseudo-label supervision, and the pseudo-labels are dynamically updated in each iteration. When the cluster assignment change of two consecutive iterations is lower than the preset threshold or the maximum number of iterations is reached, the training is terminated, and the final clustering result is determined by the category label corresponding to the highest probability in each row of the prediction matrix.

[0029] In this embodiment, a schematic diagram of the overall framework of the short text clustering method based on adaptive optimal transmission and three-level robust representation is shown below. Figure 2 As shown, the pseudocode for the short text clustering method based on adaptive optimal transmission and three-level robust representation is as follows: Algorithm 1: A Short Text Clustering Method Based on Adaptive Optimal Transmission and Three-Level Robust Representation Input: Dataset Pre-epochs (warm-up); Total epochs (training sessions) Output: Clustering model Based on dataset Dataset obtained by explicit data augmentation ; Load the pre-trained SBERT as the encoder and initialize the network. , , Obtain batch samples ; Embedded representation obtained through SBERT and embedding of the original sample Virtual data augmentation is obtained ; go through Obtain features ,go through and Probability of obtaining ; Calculate pseudo-labels using K-means ; Calculate contrast loss ; renew and Parameters in; else Calculate probability ; Calculate pseudo-labels using OT. ; Calculate total loss ; renew , , and Parameters in; end if end for.

[0030] The working principle and beneficial effects of the above technical solution are as follows: Short texts are encoded using a pre-trained language model, and sample diversity is enhanced through both explicit and virtual augmentation strategies. Then, the encoded results are input into a pseudo-label generation module, which dynamically solves the transfer matrix based on a discrete optimal transfer framework combined with entropy regularization and a distribution variable penalty function, thereby generating high-quality pseudo-labels. Next, under the guidance of the pseudo-labels, three mechanisms—prototype learning, class comparison learning, and instance comparison learning—are executed to optimize feature representations from three levels: intra-class aggregation, inter-class differentiation, and sample discrimination, respectively. Finally, through iterative training in the warm-up and joint optimization phases, closed-loop optimization of feature extraction, pseudo-label generation, and robust representation learning is achieved. Training terminates when the cluster assignment change in two consecutive iterations is lower than a preset threshold or the maximum number of iterations is reached. This invention effectively alleviates the problems of data sparsity and class imbalance in short text clustering, improving the stability and accuracy of clustering results.

[0031] Example 2: Building upon Example 1, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. It performs multi-type data augmentation processing on the semantic representation to obtain enhanced sample representations, including: Read the semantic representation of the original short text and determine the embedding vector of the original short text; The original short text is replaced with keywords to generate explicit augmented samples. At the same time, the embedded vector is randomly erased to generate virtual augmented samples.

[0032] In this embodiment, keyword replacement of the original short text includes: using a BERT pre-trained model to generate explicit augmented samples through keyword replacement. ;in, Represents the original text; This indicates enhanced display.

[0033] In this embodiment, a random erasure strategy is used to generate virtual augmented samples from the embedding vector of the original short text.

[0034] In this embodiment, display sample augmentation refers to replacing keywords in the original short text to generate new samples. Its purpose is to introduce word variations while maintaining semantic consistency, thereby enhancing the diversity of training data. Synonym substitution can simulate user expression variations and improve the model's ability to recognize texts that are semantically similar but have different wording.

[0035] In this embodiment, virtual augmented samples refer to new samples generated by performing a random erasure operation on the embedding vector of the original short text. The purpose is to artificially introduce noise or missing data at the feature representation level to enhance the model's adaptability to incomplete or noisy data.

[0036] In this embodiment, keyword replacement refers to using the BERT language model to identify and replace high-weight keywords in the original text to generate semantically similar variant text. The purpose is to expand the data distribution by explicitly modifying the text content.

[0037] The working principle and beneficial effects of the above technical solution are as follows: By combining multiple data augmentation strategies, including explicit augmentation and virtual augmentation, the performance of the short text clustering model is effectively improved; keyword replacement expands sample diversity at the semantic level and avoids over-reliance on the original vocabulary; random erasure introduces perturbation at the feature level and enhances the model's robustness to noise and missing information; this hybrid augmentation method jointly alleviates the problems of sparsity and semantic incompleteness of short text data, thereby improving the accuracy and stability of the clustering results.

[0038] Example 3: Based on Example 2, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation, including: The weight values ​​of each word in the preset vocabulary set are calculated based on the pre-trained BERT model. The preset vocabulary set is sorted according to the weight values ​​of each word. The top preset proportion of words are selected as target words to replace keywords in the original short text, generating explicit augmented samples. When performing random erasure on the embedded vector, the second preset proportion of words in the embedded vector are masked.

[0039] In this embodiment, the explicit enhanced context enhancer is based on the BERT model and achieves semantic diversity expansion by replacing 30% (i.e., the first preset proportion) of high-weight keywords in the text.

[0040] In this embodiment, virtual enhancement constructs perturbation samples by randomly masking 30% (i.e., the second preset ratio) of tokens in the original text embedding representation, while keeping the sequence length unchanged.

[0041] In this embodiment, the enhanced text is mapped to a 768-dimensional vector space, its category distribution is predicted by a clustering head, and further reduced to a 128-dimensional representation through a projection layer for subsequent contrastive learning. A schematic diagram of the feature extraction structure is shown below. Figure 3 As shown.

[0042] The beneficial effects of the above technical solutions are: they jointly alleviate the problems of sparsity and semantic incompleteness in short text data, thereby improving the accuracy and stability of clustering results.

[0043] Example 4: Based on Example 1, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. In step 2, pseudo-labels are assigned to the enhanced sample representations based on the optimal transmission algorithm, including: The original text is mapped into feature representations based on encoding networks; The feature representation is processed according to the clustering network, and the clustering probability of the sample is predicted based on the processing result. A cost matrix is ​​constructed based on the clustering assignment probability of the samples, and the migration cost from samples to categories is quantified based on the cost matrix. Construct an optimization objective, and determine the association matrix between samples and categories based on the optimization objective, wherein the optimization objective is constrained by an entropy regularization term and a distribution variable penalty function; The correlation matrix satisfying the optimization objective is dynamically calculated according to a preset iterative algorithm; Based on the association matrix, the category with the highest association strength for each sample is output, and the category with the highest association strength is used as the pseudo-label for assignment.

[0044] In this embodiment, mapping the original text to a feature representation based on an encoding network can be achieved using: an encoding network. Original text Mapping to feature representation ,in, Indicates batch size, Indicates the feature dimension; This represents the embedding matrix output by the encoder; Represents the real number field; Represents the original text In coding networks Feature representation after mapping.

[0045] In this embodiment, processing the feature representation according to the clustering network and predicting the clustering probability of the samples based on the processing result can be achieved by using a clustering network constructed through fully connected layers. Predicting cluster assignment probabilities ,in This indicates the preset total number of categories; This represents the predicted class distribution matrix.

[0046] In this embodiment, a cost matrix is ​​constructed based on the clustering assignment probabilities of the samples, and the migration cost from samples to categories is quantified based on the cost matrix. An optimization objective is constructed, and the association matrix between samples and categories is determined based on the optimization objective. The optimization objective, constrained by the entropy regularization term and the distribution variable penalty function, refers to: minimizing the cross-entropy loss to generate pseudo-labels by solving the discrete optimal transport problem. ;in, The pseudo-label is used to address the potential degradation problem under random initialization conditions. A regularization penalty for the distribution variable is introduced into the optimal transmission objective function and dynamically updated during the solution of the transmission matrix. This optimization problem can be formally represented as: ;in, Represents class distribution variables; Represents the cost matrix; Represents the transfer matrix, and It is used to characterize the sample-category relationship; and This represents hyperparameters used to adjust the balance of the optimization objective; Let represent the entropy regularization term, and Prevent excessive sparsity in solutions; Represents the distribution of class variables The penalty function, and ; Indicates transpose; Denotes the uniform distribution vector of the sample distribution; the sample distribution is set to uniform distribution. ; It is a vector consisting entirely of 1s; This represents the function for finding the minimum value.

[0047] In this embodiment, dynamically calculating the correlation matrix that satisfies the optimization objective according to the preset iterative algorithm refers to using the Lagrange multiplier algorithm to solve the optimization objective by iteratively updating the new Lagrange multiplier, the transfer matrix, and the class distribution variables; the specific process is as follows: S21: Optimize the problem using the Lagrange multiplier algorithm: ,in, , and All are Lagrange multipliers; S22: Variables Differentiating the above equation, we get: ,in, Indicates the sample index, corresponding to the transfer matrix. Row index; Represents the category index, corresponding to the transfer matrix. Column index; S23: Fixed ,because and We can obtain: ; S24: Fixed and And update using the following methods S25: S26: Variables Differentiating the above equation, we can obtain S27: From this, we can obtain the discriminant of the above equation. S28: ,because S29: Therefore, choose S210: Substitute the above equation back into the original constraint conditions The formula is defined as: ,in, It is a root of the above equation, and can be solved using Newton's method; S211 definition Then update ,Right now The iteration count is set to 10, and then obtained through S22. S212: By iteratively updating S23, S29, and S211, the transfer matrix of S22 can be obtained. .

[0048] In this embodiment, outputting the category with the highest association strength for each sample based on the association matrix, and assigning the category with the highest association strength as a pseudo-label, can be done after obtaining the transfer matrix. Then, the pseudo-labels are obtained through argmax operation, specifically: .

[0049] In this embodiment, the flowchart of the pseudo-tag generation process described above is as follows: Figure 4 As shown.

[0050] The beneficial effects of the above technical solution are: treating pseudo-label generation as a global assignment problem, rather than independently and greedily assigning labels to each sample, helps to reduce conflicts and noise; by introducing adaptive distribution variables, it is possible to avoid pre-setting strict class distributions, but rather allow the model to learn and adjust dynamically during training, effectively preventing cluster degradation; by combining entropy regularization, the assignment results are made smoother and more robust, avoiding overconfident erroneous assignments.

[0051] Example 5: Based on Example 1, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. In step 3, the corresponding loss function is calculated based on the pseudo-labels and the three-level robust representation, and the model parameters are updated in the server's CPU using the loss function, including: The three-level robust representation learning includes: execution prototype learning, class comparison learning, and instance comparison learning; Specifically, the prototype learning maintains a learnable prototype vector for each category and achieves intra-class aggregation through an improved cross-entropy loss; the class contrastive learning constrains different augmented views of the same original text to maintain consistency in the prediction distribution; and the instance contrastive learning brings the projected representations of positive sample pairs closer together and pushes the projected representations of negative sample pairs further apart in the projection space. The cumulative sum is calculated based on the loss functions of prototype learning, class comparison learning, and instance comparison learning to obtain the total loss function.

[0052] In this embodiment, a schematic diagram of the three-level robust representation learning structure is shown below. Figure 5 As shown.

[0053] Example 6: Building upon Example 5, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. The loss function for prototype learning is: ; in, Represents the prototype vector The loss function; Represents the prototype vector; Indicates batch size; This represents the sample's serial number value. Represents the logarithmic function with base e; Indicates the first Features of each sample; Indicates the first Pseudo-labels for individual samples The prototype vector; Indicates the first The pseudo-labels of the samples; Indicates the total number of categories; The index value representing the category; Indicates the first The prototype vectors corresponding to each category; This indicates temperature hyperparameters; Represents the similarity function; This represents an exponential function.

[0054] Example 7: Building upon Example 5, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. The loss function for class contrastive learning is: ; in, This represents the loss function used to predict the probability distribution of classes in contrastive learning. This represents the predicted category probability distribution value; Indicates the first One sample in virtual augmented view The predicted category probability distribution values ​​are as follows; Indicates the first One sample is displayed in the enhanced view. The predicted category probability distribution values.

[0055] Example 8: Building upon Example 5, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. The loss function for instance contrastive learning is as follows: positive sample pairs are defined as augmented samples from the same original text, and negative sample pairs are defined as augmented samples from different original texts; for all augmented samples in a batch, their projected representation is used to calculate the loss, and its loss function is: ; in, The loss function for instance-based comparative learning; Indicates comparative loss; Indicates the first Contrast loss between virtual enhancement and display enhancement for each sample; Indicates the first Contrast loss between visual enhancement and virtual enhancement for each sample.

[0056] In this embodiment, This is a virtual enhanced vector index, and This refers to the vector index used for display enhancement, calculated as follows: This is because InfoNCE loss has a direction.

[0057] In this embodiment, instance-based contrastive learning aims to bring the projected representations of positive sample pairs closer together while pushing away negative sample pairs; assuming a batch contains a total of An augmented sample, whose projection is represented as: Given a pair, where two texts are augmented from the same original text, the others... One enhanced text is considered a negative sample; a pair of positive texts is considered a negative sample. The loss is defined as: ,in, This indicates the metric; when calculating instance contrast loss, all pairs in the batch are considered, including... and ,Right now: .

[0058] Example 9: Based on Example 5, Example 6, Example 7, or Example 8, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. The method calculates the cumulative sum based on the loss functions of prototype learning, class contrastive learning, and instance contrastive learning to obtain the total loss function, including: ; in, , , For weight hyperparameters.

[0059] Example 10: Based on Example 1, this example provides a short text clustering method based on adaptive optimal transmission and three-level robust representation. In step 4, steps 1-3 are executed iteratively, and the target clustering result is output when the clustering result meets the convergence condition, including: S41: Use K-means to cluster the initial feature representations to generate initial pseudo-labels; S42: Under pseudo-label supervision, iteratively optimize the parameters of Sentence-BERT, prototype tensor, clustering head and comparison head, and dynamically update pseudo-labels in each iteration; S43: Training terminates when the change in clustering assignment results between two consecutive iterations is lower than a preset threshold or the maximum number of iterations is reached. S44: The clustering results are determined by the category label corresponding to the highest probability in each row of the prediction matrix.

[0060] In this embodiment, the initial pseudo-label refers to the category label obtained by using a traditional clustering algorithm (such as K-means) to preliminarily divide the initial feature representation of the samples before the formal iteration training of the model begins. This provides an initial supervision signal for the subsequent deep neural network training and avoids the training instability caused by completely random initialization.

[0061] In this embodiment, the prototype tensor refers to the set of learnable parameters maintained during prototype learning, where each prototype vector represents the feature center of a category; the trainable prototype vectors enable the tight aggregation of samples within a class, thereby improving the clarity of category division.

[0062] In this embodiment, the clustering head refers to the fully connected layer module at the end of the feature extraction network, which is responsible for mapping high-dimensional features to class probability distributions and converting the continuous feature space into a discrete class distribution, thus providing a basis for pseudo-label generation and loss calculation.

[0063] In this embodiment, the contrast head refers to the projection network layer used for contrastive learning, which transforms the original features into a low-dimensional space suitable for calculating similarity, and optimizes the discriminative power of positive and negative samples through feature dimensionality reduction and normalization.

[0064] In this embodiment, the prediction matrix refers to the probability matrix output after the model's forward propagation, where each element represents the probability value of the corresponding sample belonging to a specific category. The final category assignment of the sample is determined by selecting the index corresponding to the maximum probability value in each row.

[0065] The beneficial effects of the above technical solution are: improved training efficiency and result quality of the short text clustering model. First, K-means is used to generate initial pseudo-labels, providing a stable starting point for subsequent optimization and avoiding training oscillations caused by random initialization. During the iteration process, the parameters of multiple components such as Sentence-BERT, prototype tensor, clustering head, and comparison head are jointly optimized to achieve co-evolution of feature learning and cluster assignment, while dynamically updating pseudo-labels to ensure continuous optimization of the supervision signal. A dual termination condition based on a cluster assignment change threshold and a maximum number of iterations is introduced to ensure sufficient training convergence while preventing overcomputation. Finally, the category assignment is determined based on the maximum probability of the prediction matrix, making the clustering results more reliable. This ensures a smooth transition of the model from initial coarse partitioning to final fine clustering, significantly improving the accuracy and stability of short text clustering.

[0066] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A short text clustering method based on adaptive optimal transport and three-level robust representation, characterized in that, The method comprises the following steps: Step 1: In a GPU-equipped server, encode the input original short text based on a pre-trained language model to obtain a semantic representation of the original short text, and perform multi-type data augmentation processing on the semantic representation to obtain an enhanced sample representation; Step 2: Assign pseudo labels to the enhanced sample representation based on an optimal transport algorithm; Step 3: Calculate the corresponding loss function based on the pseudo labels and based on three-level robust representation learning, and update the model parameters in the CPU of the server; Step 4: Iteratively perform steps 1-3, and output the target clustering result when the clustering result meets the convergence condition.

2. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 1, characterized in that, The multi-type data augmentation processing on the semantic representation to obtain the enhanced sample representation comprises: Reading the semantic representation of the original short text to determine the embedding vector of the original short text; Performing keyword replacement on the original short text to generate explicit enhanced samples, and performing random erasure on the embedding vector to generate virtual enhanced samples.

3. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 2, characterized in that, The method comprises the following steps: Calculating the weight values of each word in the preset word set based on the pre-trained BERT model, and sorting the preset word set according to the weight values of each word, and extracting the top first preset proportion of words as target words to replace the keywords in the original short text to generate explicit enhanced samples; When performing random erasure on the embedding vector, perform masking processing on the second preset proportion of word units in the embedding vector.

4. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 1, characterized in that, In step 2, the pseudo labels are assigned to the enhanced sample representation based on the optimal transport algorithm, which comprises: Mapping the original text to a feature representation based on an encoding network; Processing the feature representation according to a clustering network, and predicting the clustering assignment probability of the sample according to the processing result; Constructing a cost matrix based on the clustering assignment probability of the sample, and quantifying the migration cost of the sample to the category based on the cost matrix; Constructing an optimization objective, and determining the association relationship matrix between the sample and the category according to the optimization objective, wherein the optimization objective is constrained by an entropy regularization term and a distribution variable penalty function; Dynamically calculating the association relationship matrix that satisfies the optimization objective according to a preset iteration algorithm; According to the association relationship matrix, output the category with the highest association strength corresponding to each sample, and assign the category with the highest association strength as the pseudo label.

5. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 1, characterized in that, In step 3, the corresponding loss function is calculated based on the pseudo labels and based on three-level robust representation learning, and the model parameters are updated in the CPU of the server, which comprises: The three-level robust representation learning comprises: performing prototype learning, class contrast learning and instance contrast learning; Wherein, the prototype learning maintains a learnable prototype vector for each category, and realizes intra-class aggregation through an improved cross-entropy loss; the class contrast learning constrains different enhanced views of the same original text to keep consistent in the predicted distribution; the instance contrast learning narrows the projection representation of positive sample pairs and pushes away the projection representation of negative sample pairs in the projection space; According to the loss function calculation of the prototype learning, the class contrast learning and the instance contrast learning, the total loss function is obtained.

6. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 5, characterized in that, The loss function of the prototype learning is: ； wherein, represents a prototype vector loss function; represents a prototype vector; represents a batch size; represents a sequence number value of a sample, represents a logarithm function with base e; represents a feature of an i th sample; represents a prototype vector of a pseudo label of an i th sample; represents a pseudo label of an i th sample; represents a pseudo label of an i th sample; represents a total number of classes; represents a sequence number value of a class; represents a prototype vector corresponding to an i th class; represents a similarity function; represents an exponential function.

7. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 5, characterized in that, The loss function of the class contrast learning is: ； in, This represents the loss function used to predict the probability distribution of classes in contrastive learning. This represents the predicted category probability distribution value; Indicates the first One sample in virtual augmented view The predicted category probability distribution values ​​are as follows; Indicates the first One sample is displayed in the enhanced view. The predicted category probability distribution values.

8. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 5, characterized in that, The loss function of the instance contrast learning is: the positive sample pair is defined as the augmented samples from the same original text, and the negative sample pair is defined as the augmented samples from different original texts; for all augmented samples in a batch, the projected representation is used to calculate the loss, and the loss function is: ； wherein, represents a loss function for instance contrastive learning; represents a contrastive loss; represents a contrastive loss between the virtual augmentation and the display augmentation for the th sample; represents a contrastive loss between the display augmentation and the virtual augmentation for the th sample.

9. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 5 or 6 or 7 or 8, characterized in that, According to the loss function calculation of the prototype learning, the class contrast learning and the instance contrast learning, the total loss function is obtained, including: ； wherein, , , are weight hyperparameters.

10. The short text clustering method based on adaptive optimal transmission and three-level robust representation according to claim 1, characterized in that, In step 4, steps 1-3 are iteratively executed, and the target clustering result is output when the clustering result meets the convergence condition, including: S41: K-means is used to cluster the initial feature representation to generate initial pseudo labels; S42: Under the supervision of the pseudo labels, the parameters of Sentence-BERT, the prototype tensor, the clustering head and the contrast head are iteratively optimized, and the pseudo labels are dynamically updated in each iteration; S43: When the change of the clustering assignment result between two consecutive iterations is lower than a preset threshold or the maximum iteration number is reached, the training is terminated; S44: The clustering result is determined by the class label corresponding to the maximum probability of each row in the prediction matrix.

Images