A web short text data stream clustering method based on contrastive learning
By employing a contrastive learning-based approach, this study utilizes Sentence-Bert and encoder models to perform feature representation and clustering of short text data streams. This addresses the issues of insufficient semantic information capture and feature drift in short text stream clustering, achieving higher-precision clustering results. It is applicable to practical fields such as event tracking and news recommendation.
Patent Information
- Application Number
- CN202311151489.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-09-07
AI Technical Summary
Existing short text stream clustering methods cannot effectively capture semantic information, cannot cope with feature drift and concept evolution, and do not make full use of labeled data to improve clustering accuracy.
A contrastive learning-based approach is adopted to divide short text data streams into labeled and unlabeled data blocks. The Sentence-Bert pre-trained language model is used for feature representation, and the encoder is trained through a supervised contrastive loss function. After dimensionality reduction encoding, pseudo-labels are used to update the encoder model to adapt to the data distribution changes in the data stream environment, and the clusters are updated in real time.
It improves the clustering effect of short text data streams, can adapt to data changes in a timely manner, and produces more accurate and reliable clustering results, making it suitable for semi-supervised data stream environments.
Smart Images
Figure CN117235550B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of practical application in short text Web data stream mining, and particularly relates to a semi-supervised data stream clustering method based on comparative learning. BACKGROUND
[0002] With the advent of the era of big data, short text data has become one of the most important information resources in today's society. A large amount of short text data is generated on network social platforms such as microblog, Twitter, Facebook, etc. every day. These data not only contain people's personal feelings and ideas, but also cover information published by various entities such as enterprises and media. Compared with traditional texts, these data texts usually lack long context information, such as texts published on social media platforms such as Sina Weibo and Twitter, which limit the number of characters or words, making the text length more concise and compact.
[0003] The rapid growth of short text quantity brings convenience to information acquisition, but because its growth rate far exceeds the processing efficiency of people, it is difficult to effectively obtain effective information when facing a large amount of information, which causes a huge waste of time, money and energy for the general public and institutions. In addition, due to the lack of rich context and semantic information in short texts, their processing and analysis become more complex and difficult. At the same time, the mobility and real-time nature of these short text data make them have high value in various application scenarios. For example, event tracking is a method of using short text data to monitor social hot events in real time. Hot topic detection can detect the hottest topics and concerns in the current society by analyzing short text data. News recommendation is to recommend news information of interest to users by analyzing short text data. Therefore, short text clustering, as a solution to these applications, has also received more and more attention. The purpose of short text clustering is to organize a large amount of short text data into meaningful categories to better understand and use these data.
[0004] Challenge one: The high-dimensional sparsity and term ambiguity of short texts pose problems and challenges to the processing of short texts. Existing short text stream clustering methods can only capture word frequency and grammar information but cannot capture semantic information, making short text stream clustering unable to achieve a good effect.
[0005] Challenge two: In the short text data stream environment, feature drift and concept evolution caused by the development of topics are difficult to detect, resulting in low clustering accuracy.
[0006] The third challenge is that the presence of labeled data can greatly improve the performance of short text stream clustering. However, existing short text stream clustering methods do not use these labeled information as an aid to improve the clustering effect. Therefore, how to make full use of these labeled data while also being able to cope with the changes in the characteristics of unlabeled data has always been an important issue in the field of short text clustering research. Summary of the Invention
[0007] To address the aforementioned problems, this invention provides a web short text data stream clustering method based on contrastive learning. This method aims to cluster data in short text data stream environments, process short texts in real time, adapt to potential concept drift in data stream environments, and improve clustering accuracy based on labeled information. This allows for rapid detection of public opinion changes, enabling the identification and tracking of the most popular topics and concerns in society, providing enterprises or governments with the most relevant information and events, and offering more effective decision support.
[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:
[0009] The present invention provides a web short text data stream clustering method based on contrastive learning, characterized by the following steps:
[0010] Step 1: Divide the given short text data stream D into two data blocks according to whether it contains labels, denoted as D = {D...} l D u}, where D l D u These represent labeled and unlabeled data blocks in the short text data stream D, respectively. Where m represents the labeled data block D l The number of short texts, where n represents the total number of short texts in the short text data stream D, and m < n; and These respectively represent the labeled data blocks D l The i-th short text and the unlabeled data block D u The j-th short text is given, 1≤i≤m, 1≤j≤nm; and we have: and These represent labeled data blocks D. l The i-th short text The feature space, and the unlabeled data block D u The feature space of the j-th short text, and This indicates the labeled data block D. l The i-th short text The r-th word, and This indicates the labeled data block D. l The i-th short text Length; This indicates the unlabeled data block D u The j-th short text The p-th word, and This indicates the unlabeled data block D u The j-th short text Length; This indicates the labeled data block D. l The i-th short text The class tag space, and has: K represents the number of class tags in the class tag space. This indicates the labeled data block D. l The i-th short text Class tag space The k-th dimension class label value, 1≤k≤K, Thus, the labeled data block D is obtained. l Class tag space set
[0011] Step 2: Based on the labeled data block D l Train the encoder and initialize the cluster set and similarity threshold:
[0012] Step 2.1: Use the pre-trained language model Sentence-Bert to analyze the feature space. Processing is performed to obtain the i-th short text. Feature representation in, Represents the i-th short text Feature representation The s-th feature value in the text, 1≤s≤S, where S represents the ith short text. Feature representation The vector dimension; thus obtaining the feature representation set composed of the feature representations of m short texts.
[0013] Step 2.2: Utilize the labeled data block D l The encoder is trained using the feature representation set V and the class label space set Y, and the supervised contrastive loss function shown in Equation (1) is minimized. This yields the trained encoder model Encoaer and its model parameters θ;
[0014]
[0015] In equation (1), A(i) represents the short text excluding the i-th short text in the feature representation set V. Feature representation The feature representation set of other short texts besides the i-th short text, P(i) represents the feature representation set V containing the i-th short text. Feature representation In addition, with the i-th short text Class tag space The same set of feature representations for other short texts, |P(i)| represents the number of short texts in the feature representation set P(i); τ represents a constant temperature parameter, τ∈R. + ;R + Represents positive real numbers; Let P(i) represent the feature representation of the b-th short text in the feature representation set. Let A(i) represent the feature representation of the a-th short text in the feature representation set A(i).
[0016] Step 2.3: Use the encoder model (Encoder) to process the i-th short text. Feature representation Dimensionality reduction encoding is performed to obtain the i-th short text. Dimensionality reduction feature representation This yields the dimensionality-reduced labeled data block D′. l ;
[0017] Step 2.4: Define the current time as t; define the index of the current cluster as k;
[0018] Step 2.5: Initialize t = 0;
[0019] Step 2.6: Initialize k = 1;
[0020] Step 2.7: Initialize i = 1;
[0021] Step 2.8: Determine if i ≤ m is true. If true, proceed to step 2.9; otherwise, it indicates that the dimensionality-reduced labeled data block D′ at time t has been processed. l Extract all short texts and obtain labeled data block D′. l The set of clusters at time t. 1≤k≤m′, m′≤m; and execute step 2.12; where m′ represents the total number of clusters at time t;
[0022] Step 2.9: Determine the i-th short text at time t. tags If yes, go to step 2.10; otherwise, go to step 2.11.
[0023] Step 2.10, initialize the kth clustering cluster at time t according to formula (2) Then, let i+1 be assigned to i, and go to step 2.8.
[0024]
[0025] In formula (2), respectively represent the number of short texts, the short text set, the clustering center and the update time of the kth clustering cluster at time t
[0026] Step 2.11, obtain the kth clustering cluster at time t according to formula (3) Let k+1 be assigned to k, and i+1 be assigned to i, and go to step 2.8.
[0027]
[0028] In formula (3), <- represents an assignment symbol.
[0029] Step 2.12, select the dimension-reduced labeled data block D' l Any two short texts in the dimension-reduced labeled data block D' 1≤p, q≤m, p≠q, and The class label space of is the same as The class label space of
[0030] According to formula (4), calculate the cosine similarity of any two short texts in the dimension-reduced labeled data block D' l with the same class label space and
[0031]
[0032] In formula (4), respectively represent the modulus of the dimension-reduced feature representation of the pth short text l in the labeled data block D and the modulus of the dimension-reduced feature representation of the qth short text
[0033] Step 2.13, repeat step 2.12 until the dimension-reduced labeled data block D' l the similarity between all short texts with the same label in D and calculate the average value to obtain
[0034] Step 2.14, select the labeled data block D' after dimensionality reduction l any two short texts with different class labels in D 1≤p', q'≤m, p'≠q', and the class label space of is not the same as the class label space of ;
[0035] According to formula (5), calculate the labeled data block D' after dimensionality reduction l any two short texts with different class label spaces and the cosine similarity of
[0036]
[0037] In formula (5), respectively represent the modulus of the dimensionality reduction feature representation of the p'th short text in the labeled data block D' and the modulus of the dimensionality reduction feature representation of the q'th short text in the labeled data block D' ;
[0038] Step 2.15, repeat step 2.14 until the labeled data block D' after dimensionality reduction is calculated l the similarity between all short texts with different labels in D and calculate the average value to obtain
[0039] Step 2.16, calculate the threshold R according to formula (6);
[0040]
[0041] In formula (6), β represents a constant weight, 0≤β≤1;
[0042] Step 3: cluster the unlabeled data block D u and update the cluster set and the encoder:
[0043] Step 3.0, define the current period as T, and initialize T=0; the encoder model Encoder is taken as the encoder model Encoder of the T period T ; the encoder parameter θ is taken as the model parameter α T of the T period;
[0044] Step 3.1: Initialize j = t + 1, and set the unlabeled data block D... u The j-th short text As a short text at time t+1
[0045] Using the pre-trained language model Sentence-Bert and the encoder for the T-th time period T right Feature extraction is performed to obtain the dimensionality-reduced feature representation at time t+1.
[0046] Step 3.2: Calculate the feature representation after dimensionality reduction at time t+1 according to equation (7). The cluster set Z at time t is respectively compared with the cluster set Z at time t. t Calculate the cosine similarity of all clusters in the dataset, and take the maximum cosine similarity at time t+1.
[0047]
[0048] In equation (7), express and Cosine similarity;
[0049] Step 3.3, if If the condition is met, proceed to step 3.4; otherwise, proceed to step 3.5.
[0050] Step 3.4: Extract the short text at time t+1. Allocated to The corresponding cluster Thus, the max-th cluster at time t+1 can be obtained using equation (8). And based on the index of the cluster to which it was assigned, the short text at time t+1. Max with pseudo-tags:
[0051]
[0052] In equation (8), These represent the max-th cluster at time t+1. The number of short texts, the dimensionality-reduced short text set, cluster centers and update time, and have Let each represent the max-th cluster at time t. The number of short texts, dimensionality reduction of short text sets, and cluster centers;
[0053] Step 3.5: Assign m′+1 to m′, and generate a new cluster according to equation (9). This updates the cluster set Z at time t. t The set of clusters at time t+1 And provide the short text for time t+1. Add the pseudo-label m′;
[0054]
[0055] In equation (9), These represent the m′-th cluster at time t+1, respectively. The number of short texts, the dimensionality-reduced short text set, cluster centers, and update time;
[0056] Step 3.6: Update the cluster set Z at time t+1 according to equation (10). t+1 The weight of the kth cluster in
[0057]
[0058] In equation (10), Z represents the set of clusters at time t+1. t+1 The update time of the k-th cluster in the algorithm, where c represents the correction parameter;
[0059] Step 3.7 If the cluster size is below a given threshold l, then delete the cluster set Z at time t+1. t+1 If the k-th cluster is found, then the set of clusters Z at time t+1 is retained; otherwise, the set of clusters Z is retained. t+1 The k-th cluster in the middle;
[0060] Step 3.8: After assigning t+1 to t, determine whether t%U=0 is true. If it is true, proceed to step 3.9; otherwise, return to step 3.1 and execute sequentially. Here, % represents the modulo operator and U represents the short text quantity threshold.
[0061] Step 3.9: Use the feature representations and pseudo-labels of the U unlabeled data points to train the encoder for time period T. T Train the encoder model and obtain the encoder model for time period T+1. T+1 and its model parameter α T+1 ;
[0062] Step 3.10: According to equation (11), the model parameters θ for the (T+1)th time period are... T+1 The updated model parameters θ′ are obtained by performing the update. T+1 ;
[0063] θ′ T+1 ←ω·θ′ T +(1-ω)θ T+1 (11)
[0064] In formula (11), ω represents a momentum coefficient, and ω [0, 1); θ' represents an updated model parameter of a T period, and when T = 0, θ' = θ T T T
[0065] After step 3.11, T+1 is assigned to T, and the step 3.1 is executed in sequence.
[0066] The electronic device comprises a memory and a processor, and the memory is used for storing a program supporting the processor to execute the Web short text data stream clustering method, and the processor is configured to execute the program stored in the memory.
[0067] The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the Web short text data stream clustering method are executed.
[0068] Compared with the prior art, the present application has the following beneficial effects:
[0069] 1、The present application first converts short text into feature representation by using the Sentence-Bert pre-training language model, models the data through label information, trains an encoder suitable for the current data distribution, minimizes the supervised contrast loss function, reduces the vector dimension of the feature representation through the encoder model, and labels the unlabelled data in the clustering process with pseudo labels, and updates the parameters of the encoder model using the pseudo label information to adapt to the data distribution changes in the data stream environment. Compared with the existing feature representation method, the present application is more suitable for feature representation learning in a semi-supervised data stream environment, and improves the feature representation effect in a semi-supervised short text data stream environment.
[0070] 2、The present application updates the clustering cluster in real time and deletes the outdated cluster in order to cope with the infinite length and concept evolution in the short text stream. The method can adapt to data changes in time and produce more accurate and reliable clustering results.
[0071] 3、The present application solves the important practical problem of short text data stream clustering, and the experimental results on two real data sets and two synthetic data sets show that the method can effectively improve the clustering effect of short text data stream. The present application has important application value in the fields of event tracking, hot topic detection, news recommendation, etc. BRIEF DESCRIPTION OF DRAWINGS
[0072] Figure 1 It is a schematic diagram of the short text data stream clustering method framework of the present application;
[0073] Figure 2 This is a schematic diagram illustrating the process of training the encoder and initial clustering based on label information according to the present invention;
[0074] Figure 3 This is a schematic diagram illustrating the process of clustering unlabeled short texts based on cosine similarity according to the present invention. Detailed Implementation
[0075] In this embodiment, a web short text data stream clustering method based on contrastive learning is described, such as... Figure 1 As shown, it includes the following steps:
[0076] Step 1: Divide the given short text data stream D into two data blocks according to whether it contains labels, denoted as D = {D...} l D u}, where D l D u Let represent labeled and unlabeled data blocks in the short text data stream D, respectively. Where m represents labeled data block D l The number of short texts, where n represents the total number of short texts in the short text data stream D, and m < n; and These represent labeled data blocks D. l The i-th short text and the unlabeled data block D u The j-th short text is given, 1≤i≤m, 1≤j≤nm; and we have: and These represent labeled data blocks D. l The i-th short text The feature space, and the unlabeled data block D u The feature space of the j-th short text, and This indicates that there is a labeled data block D. l The i-th short text The r-th word, and This indicates that there is a labeled data block D. l The i-th short text Length; Represents unlabeled data block D u The j-th short text The p-th word, and 1≤p≤ Represents unlabeled data block D u The j-th short text Length; This indicates that there is a labeled data block D. l The i-th short text The class tag space, and has: K represents the number of class tags in the class tag space. This indicates that there is a labeled data block D. l The i-th short text class tag space The k-th dimension class label value, 1≤k≤K, This results in labeled data block D. l Class tag space set
[0077] In this embodiment, as shown in Table 1, given a short text data stream containing 10 short texts, two data blocks D = {D} are divided based on tag information. l D u};
[0078] Table 1 shows the data blocks divided into a short text data stream.
[0079]
[0080] Step 2: As Figure 2 As shown, based on labeled data block D l Train the encoder and initialize the cluster set and similarity threshold:
[0081] Step 2.1: Use the pre-trained language model Sentence-Bert to analyze the feature space. Processing is performed to obtain the i-th short text. Feature representation in, Represents the i-th short text Feature representation The s-th eigenvalue in the text, 1≤s≤S, where S represents the i-th short text. Feature representation The vector dimension; thus obtaining the feature representation set composed of the feature representations of m short texts.
[0082] Step 2.2: Utilize labeled data block D l The feature representation set V and the class label space set Y are used to train an encoder model with two multilayer perceptrons. Equation (12) is used as the loss function of the model, the batch size is set to 1, the pre-trained word embedding dimension is set to 768, the hidden layer size is set to 512, the number of iterations is set to 5, and the learning rate is set to 0.0005. The trained encoder model Encoder and its model parameters θ are obtained.
[0083]
[0084] In equation (1), A(i) represents the short text excluding the i-th short text in the feature representation set V. Feature representation The feature representation set of other short texts besides the i-th short text, P(i) represents the feature representation set V containing the i-th short text. Feature representation In addition, with the i-th short text Class tag space The same set of feature representations for other short texts, |P(i)| represents the number of short texts in the feature representation set P(i); τ represents a constant temperature parameter, τ∈R. + ;R + Represents positive real numbers; Let P(i) represent the feature representation of the b-th short text in the feature representation set. Let A(i) represent the feature representation of the a-th short text in the feature representation set A(i).
[0085] Step 2.3: Use the encoder model (Encoder) to process the i-th short text. Feature representation Dimensionality reduction encoding is performed to obtain the i-th short text. Dimensionality reduction feature representation This yields the dimensionality-reduced labeled data block D′. l ;
[0086] Step 2.4: Define the current time as t; define the index of the current cluster as k;
[0087] Step 2.5: Initialize t = 0;
[0088] Step 2.6: Initialize k = 1;
[0089] Step 2.7: Initialize i = 1;
[0090] Step 2.8: Determine if i ≤ m is true. If true, proceed to step 2.9; otherwise, it means that the labeled data block D′ after dimensionality reduction at time t has been processed. l Extract all short texts and obtain labeled data block D′. l The set of clusters at time t. 1≤k≤m′,m′≤m; and execute step 2.12; where m′ represents the total number of clusters at time t.
[0091] Step 2.9: Determine the i-th short text at time t. tags Check if the condition is met. If it is met, proceed to step 2.10; otherwise, proceed to step 2.11.
[0092] Step 2.10: Initialize the k-th cluster at time t according to equation (13). Then, assign i+1 to i and execute step 2.8;
[0093]
[0094] In equation (2), They represent the k-th cluster at time t, respectively. The number of short texts, the set of short texts, the cluster centers, and the update time, and have
[0095] Step 2.11: Obtain the k-th cluster at time t according to equation (14). Assign k+1 to k, and i+1 to i, then proceed to step 2.8:
[0096]
[0097] In equation (3), ← represents the assignment symbol.
[0098] Step 2.12: Select the labeled data block D′ after dimensionality reduction. l Any two short texts with the same class tag 1≤p, q≤m, p≠q, and Class tag space and Class tag space same;
[0099] The dimension-reduced labeled data block D′ is calculated according to equation (15). l Any two short texts with the same class tag space and cosine similarity
[0100]
[0101] In equation (15), These represent labeled data blocks D. l The p-th short text Dimensionality reduction feature representation in The modulus and the qth short text Dimensionality reduction feature representation The model.
[0102] Step 2.13: Repeat step 2.12 until the dimensionality-reduced labeled data block D′ at time t is calculated. lthe similarity between all the short texts with the same label in the labeled data block D and the average value is obtained
[0103] Step 2.14, selecting the labeled data block D' after dimension reduction l the similarity between any two short texts with different class labels 1≤p', q'≤m, p'≠q', and the class label space of is not the same as the class label space of ;
[0104] According to formula (16), the labeled data block D' after dimension reduction is calculated l any two short texts with different class label spaces and the cosine similarity of
[0105]
[0106] In formula (16), respectively represent the modulus of the dimension-reduced feature representation of the p'th short text in the labeled data block D' and the modulus of the dimension-reduced feature representation of the q'th short text .
[0107] Step 2.15, repeating step 2.14 until the labeled data block D' after dimension reduction is calculated l the similarity between all the short texts with different labels in the labeled data block D' and the average value is obtained
[0108] Step 2.16, calculating the threshold R according to formula (17);
[0109]
[0110] In formula (17), β represents a constant weight, 0≤β≤1, and β is 0.8.
[0111] In this embodiment, the encoder is trained according to the labeled data block as shown in Table 2, and the cluster set and the similarity threshold are initialized.
[0112] (1) Table 2 gives a set of text data stream containing 10 short texts, according to the text after stop words by Sentence-Bert model, get the quantized 768-dimensional feature representation set, as shown in Table 3, and then train the encoder model according to the label information and feature representation set. For the encoder model, it is composed of two layers of multilayer perception, the dimension of the hidden layer is set to 512, the iteration number is set to 5, and the learning rate is set to 0.0005;
[0113] (2) For the label information 4 of the first short text in the labeled data block D l , because the label information 4 does not appear, a new cluster C1 is generated, and C1 is initialized as {1, {1}, [-1.83308393e-01 3.21923465e-01 -1.26075596e-01 2.27983952e-01...}, 0}, and the mapping relationship between the label and the cluster is recorded as 4->1;
[0114] (3) For the label information 2 of the second short text in the labeled data block D l , because the label information 2 does not appear, a new cluster C2 is generated, and C2 is initialized as {1, {2}, [-5.22407070e-02 -2.15709761e-01 -1.67349845e-01 -1.27932623e-01...}, 0}, and the mapping relationship between the label and the cluster is recorded as 2->2;
[0115] (4) For the label information 2 of the seventh short text in the labeled data block D l , 2 has appeared, and the label 2 corresponds to the cluster 2, so the cluster C2 is updated, C2 is {2, {2, 7}, [(-5.22407070e-02+8.08653355e-01) / 2, (-2.15709761e-01+1.11740110e-02) / 2, (-1.67349845e-01+8.98510069e-02) / 2, (-1.27932623e-01+1.26502514e-01) / 2...], 0};
[0116] (5) After processing all the short texts in the labeled data block D l according to steps 2.6-2.8, the initial cluster set is obtained;
[0117] (6) Initialize num=0, sum=0;
[0118] (7) Take any one short text from cluster 1, take other short texts from cluster 1, and calculate the cosine similarity of the two short texts Counter num = num + 1 = 1, similarity sum sum = sum + d = 0.62;
[0119] (8) Take another short text from cluster 1, calculate their cosine similarity similarity sum sum = sum + d = 0.62 + 0.73 = 1.35;
[0120] (9) Calculate the similarity of any two different short texts in cluster 1 according to formula (15) until the calculation is completed;
[0121] (10) Select the cluster that has not been processed to calculate the similarity, until all clusters are calculated;
[0122] (11) Calculate
[0123] (12) Reset sum = 0, num = 0;
[0124] (13) Take any short text from cluster 1, take a short text from other clusters except cluster 1, calculate their cosine similarity Counter num = num + 1 = 1, similarity sum sum = sum + d = -0.34;
[0125] (14) Calculate the similarity of any short text in cluster 1 with any short text in other clusters according to formula (16), until all short texts in other clusters are calculated;
[0126] (15) Select the cluster that has not been processed to calculate the similarity, until all clusters are calculated;
[0127] (16) Calculate
[0128] (17) The parameter β of the similarity threshold is 0.8, the similarity threshold is calculated as R = 0.8 x 0.8146 + 0.2 x (-0.465) = 0.56;
[0129] Table 2 10 sample data in a certain short text data stream
[0130]
[0131]
[0132] Table 3 Text vector set after vectorization of 10 sample data
[0133]
[0134]
[0135] Step 3: Clustering cluster partition is performed on the unlabeled data block D u , and the cluster set and the encoder are updated:
[0136] Step 3.0, define the current period as T, and initialize T = 0; the encoder model Encoder is taken as the encoder model Encoder of the T period T ; the encoder parameter θ is taken as the model parameter θ T of the T period;
[0137] Step 3.1, initialize j = t + 1, and take the jth short text in the unlabeled data block D u as the short text at the t + 1 moment
[0138] Feature extraction is performed on T by using the pre-trained language model Sentence-Bert and the encoder Encoder of the T period, to obtain the dimension-reduced feature representation at the t + 1 moment
[0139] Step 3.2, calculate the cosine similarity between the dimension-reduced feature representation at the t + 1 moment and all the cluster sets in the cluster set Z t at the t moment respectively, and take the maximum value of the cosine similarity at the t + 1 moment
[0140]
[0141] In formula (18), denotes the cosine similarity between and ;
[0142] Step 3.3, if , step 3.4 is performed; otherwise, step 3.5 is performed.
[0143] Step 3.4, the short text at the t + 1 moment is divided into the cluster corresponding to , so as to obtain the maxth cluster at the t + 1 moment by using formula (19), and the short text at the t + 1 moment is given a pseudo-label max according to the subscript max of the cluster to which it is divided:
[0144]
[0145] In equation (19), These represent the max-th cluster at time t+1. The number of short texts, the dimensionality-reduced short text set, cluster centers and update time, and have Let each represent the max-th cluster at time t. The number of short texts, dimensionality reduction of short text sets, and cluster centers.
[0146] Step 3.5: Assign m′+1 to m′, and generate a new cluster according to equation (20). This updates the cluster set Z at time t. t The set of clusters at time t+1 And provide the short text for time t+1. Add the pseudo-label m′;
[0147]
[0148] In equation (20), These represent the m′-th cluster at time t+1, respectively. The number of short texts, the dimensionality-reduced short text set, the cluster center, and the update time.
[0149] Step 3.6: Update the cluster set Z at time t+1 according to equation (21). t+1 The weight of the kth cluster in
[0150]
[0151] In equation (21), Z represents the set of clusters at time t+1. t+1 The update time of the k-th cluster, where c represents the denominator correction parameter, c = 10. -5 .
[0152] Step 3.7 If the cluster size is below a given threshold l (l = 0.0005), then delete the cluster set Z at time t+1. t+1 If the k-th cluster is found, then the set of clusters Z at time t+1 is retained; otherwise, the set of clusters Z is retained. t+1 The k-th cluster in the middle.
[0153] Step 3.8: After assigning t+1 to t, determine whether t%U=0 is true. If it is true, proceed to step 3.9; otherwise, return to step 3.1 and execute sequentially. Here, % represents the remainder symbol, and U represents the short text quantity threshold, U=2000.
[0154] Step 3.9, using the feature representation of the U pieces of unlabeled data and the pseudo label to train the encoder Encoder of the T-th period T Training is performed, and an encoder model Encoder of the T+1-th period is obtained T+1 and its model parameters θ T+1 .
[0155] Step 3.10, updating the model parameters θ T+1 of the T+1-th period according to formula (22) to obtain updated model parameters θ' T+1 ;
[0156] θ' T+1 ← ω· θ' T + (1-ω)θ T+1 (22)
[0157] In formula (22), ω represents a momentum coefficient, and ω ∈ [0, 1); θ' T represents the updated model parameters of the T-th period, and when T = 0, let θ' T = θ T .
[0158] Step 3.11, after assigning T+1 to T, return to step Step 3.1 for sequential execution.
[0159] As shown in Figure 3 , in this example, for a piece of incoming unlabeled short text data, its belonging is calculated according to its vectorized representation, and then the clustering cluster and the encoder model are updated:
[0160] (1) for the first incoming short text, set the current timestamp t = 0;
[0161] (2) remove stop words according to the stop word table;
[0162] (3) obtain an initial text vector according to a pre-trained model;
[0163] (4) perform feature dimension reduction and conversion according to the encoder to obtain a converted feature vector;
[0164] (5) initialize max t+1 = -1,
[0165] (6) take the first clustering cluster in the clustering cluster set, and calculate the cosine similarity between the feature vector of the short text and the clustering cluster center vector representation
[0166] (7) because 0.32 > -1, max t+1 = 0.32,
[0167] (8) Repeat step (6) and step (7) until all the clustering clusters are traversed, and get max t+1 = 0.75,
[0168] (9) Because 0.75 > 0.56, the short text is attributed to the clustering cluster 4, and the clustering cluster is updated
[0169] (10) Take the first clustering cluster of the clustering cluster set as the last update time, calculate the clustering cluster weight Because 1 > 0.0005, the clustering cluster is not updated;
[0170] (11) Repeat step (10) for the clustering cluster which has not calculated the clustering cluster weight until all the clustering clusters are iterated.
[0171] Repeat step (1) to step (11) until the processed unlabeled data reaches 2000. Re-train the encoder Encoder T Get the encoder Encoder T+1 and the parameter θ T+1 , and update the encoder parameter θ T+1 according to formula (22).
[0172] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0173] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to execute the steps of the above method.
Claims
1. A Web short text data stream clustering method based on contrastive learning, characterized in that, is performed as follows: Step 1: Divide the given short text data stream D into two data blocks according to whether it contains labels, denoted as D = {D...} l D u }, where D l D u These represent labeled and unlabeled data blocks in the short text data stream D, respectively. Where m represents the labeled data block D l The number of short texts, n represents the total number of short texts in the short text data stream D, and m represents the number of short texts. <n; and These respectively represent the labeled data blocks D l The i-th short text and the unlabeled data block D u The j-th short text is given, 1≤i≤m, 1≤j≤nm; and we have: and These represent labeled data blocks D. l The i-th short text The feature space, and the unlabeled data block D u The feature space of the j-th short text, and This indicates the labeled data block D. l The i-th short text The r-th word, and This indicates the labeled data block D. l The i-th short text Length; This indicates the unlabeled data block D u The j-th short text The p-th word, and This indicates the unlabeled data block D u The j-th short text Length; This indicates the labeled data block D. l The i-th short text The class tag space, and has: K represents the number of class tags in the class tag space. This indicates the labeled data block D. l The i-th short text Class tag space The k-th dimension class label value, 1≤k≤K, Thus, the labeled data block D is obtained. l Class tag space set Step 2: Based on the labeled data block D l Train the encoder and initialize the set of clustering clusters and the similarity threshold: Step 2.
1. Process the feature space with a pre-trained language model Sentence-Bert to obtain the feature representation of the i-th short text wherein, represents the s-th feature value in the feature representation of the i-th short text , 1≤s≤S, S represents the vector dimension of the feature representation of the i-th short text ; thereby obtaining a feature representation set composed of the feature representations of the m short texts Step 2.2, training the encoder with the labeled data blocks D l characteristic representation set V and class label space set Y, and minimizing the supervised contrast loss function as shown in equation (1) to obtain the trained encoder model Encoder and its model parameters θ; In equation (1), A(i) represents the short text excluding the i-th short text in the feature representation set V. Feature representation The feature representation set of other short texts besides the i-th short text, P(i) represents the feature representation set V containing the i-th short text. Feature representation In addition, with the i-th short text Class tag space The same set of feature representations for other short texts, |P(i)| represents the number of short texts in the feature representation set P(i); τ represents a constant temperature parameter, τ∈R. + ;R + Represents positive real numbers; Let P(i) represent the feature representation of the b-th short text in the feature representation set. Let A(i) represent the feature representation of the a-th short text in the feature representation set A(i). Step 2.
3. Dimensionality reduction encoding is performed on the feature representation of the ith short text by using an encoder model Encoder, to obtain a dimensionality-reduced feature representation of the ith short text , thereby obtaining the dimensionality-reduced labeled data block D' l ; Step 2.4, defining the current time as t; defining the serial number of the current clustering cluster as k; Step 2.5, initializing t = 0; Step 2.6, initializing k = 1; Step 2.7, initializing i = 1; Step 2.8, judge whether i≤m is true, if true, execute step 2.9; otherwise, represent that the dimension-reduced labeled data block D' of t moment is processed l All short texts in the middle, and get the labeled data block D' l The cluster set of t moment divided And execute step 2.12; wherein m' represents the total number of cluster of t moment Step 2.
9. At time t, determine whether the i-th short text satisfies the label , if yes, go to Step 2.10; otherwise, go to Step 2.
11. Step 2.
10. Initialize the kth cluster at time t according to formula (2) After that, assign i+1 to i, and execute step 2.
8. In formula (2), respectively represent the number of short texts, the short text set, the cluster center and the update time of the kth cluster at time t respectively represent the number of short texts, the short text set, the cluster center and the update time of the kth cluster at time t Step 2.
11. Obtain the kth cluster at time t according to formula (3) Let k+1 be assigned to k, i+1 be assigned to i, and perform step 2.8: In formula (3), <- represents an assignment symbol; Step 2.12, selecting the dimension-reduced labeled data block D' l Any two short texts in the class label space of And The class label space of And The class label space of Is the same. The dimensionality-reduced labeled data block D' is calculated according to formula (4) l Any two short texts with the same class label space And Cosine similarity In formula (4), respectively represent the labeled data block D l the pth short text in the labeled data block D the dimension-reduced feature representation of the qth short text in the labeled data block D the dimension-reduced feature representation of the qth short text in the labeled data block D the dimension-reduced feature representation of the qth short text in the labeled data block D the dimension-reduced feature representation of the qth short text in the labeled data block D Step 2.
13. Repeat Step 2.12 until the dimension-reduced labeled data block D' at time t is calculated l The similarity between all short texts with the same label in the middle is calculated, and the average value is obtained Step 2.14, selecting the labeled data block D' after dimension reduction l Any two short texts in the class label space 1≤p',q'≤m, p'≠q', and The class label space of The class label space of The class label space of Is not the same; The dimension-reduced labeled data block D' is calculated according to formula (5) l Any two short texts with different class label spaces And Cosine similarity In formula (5), respectively represent the modulus of the dimension-reduced feature representation of the p'th short text in the label data block D' and the modulus of the dimension-reduced feature representation of the q'th short text in the label data block D' . Step 2.
15. Repeat Step 2.14 until all the dimension-reduced labeled data blocks D' are computed l The similarity between all the short texts with different labels in the middle, and the average value is obtained Step 2.16, calculating the threshold R according to formula (6); In formula (6), β represents a constant weight, 0 ≤ β ≤ 1; Step 3: Clustering the data blocks D u performing cluster partitioning and updating the cluster set and the encoder: Step 3.0, define the current time period as T and initialize T = 0; set the encoder model Encoder as the encoder model Encoder for the Tth time period T ; set the encoder parameters θ as the model parameters θ for the Tth time period T ; Step 3.1, initialize j = t + 1, and the jth short text in the unlabeled data block D u as the short text at time t + 1 as the short text at time t + 1 using a pre-trained language model Sentence-Bert and an encoder Encoder of the Tth period T performing feature extraction to obtain a feature representation of the t+1th period after dimension reduction performing feature extraction to obtain a feature representation of the t+1th period after dimension reduction Step 3.
2. Calculate the reduced dimension feature representation at time t+1 according to formula (7) cosine similarity of all clustering clusters in the set Z t cosine similarity of all clustering clusters in the set Z In formula (7), denotes a cosine similarity with a cosine similarity with Step 3.
3. If Step 3.4 is performed; otherwise, Step 3.5 is performed. Step 3.4, short text at time t+1 assigned to the corresponding cluster Thus, the maxth cluster at time t+1 is obtained by using formula (8) and the short text at time t+1 is pseudo-labeled max according to the index max of the cluster to which it is assigned In equation (8), These represent the max-th cluster at time t+1. The number of short texts, the dimensionality-reduced short text set, cluster centers and update time, and have Let each represent the max-th cluster at time t. The number of short texts, dimensionality reduction of short text sets, and cluster centers; Step 3.
5. Let m'+1 be assigned to m' and generate a new cluster according to equation (9) Thus, update the cluster set Z at time t t For the cluster set Z at time t+1 And for the short text at time t+1 Tagged with pseudo-label m' In formula (9), respectively represent the number of short texts of the m'th cluster at time t+1 dimensionality reduction short text set, cluster center and update time; Step 3.
6. Update the set of cluster Z at time t+1 according to formula (10) t+1 the kth cluster weight in Z In formula (10), Zt+1= Zt+ ΔZt t+1 the update time of the kth cluster in the cluster set Zt, and c represents a correction parameter. Step 3.7、 if the given threshold l is lower than t+1, then delete the clustering cluster set Z at t+1 t+1 if the given threshold l is lower than t+1, then delete the clustering cluster set Z at t+1 t+1 if the given threshold l is lower than t+1, then delete the clustering cluster set Z at t+1 Step 3.8, after assigning t + 1 to t, judging whether t % U = 0 is established, if established, executing step 3.9; otherwise, returning to step 3.1 for sequential execution, wherein % represents a remainder symbol, and U represents a short text quantity threshold; Step 3.
9. Training the encoder Encoder of the T-th period using the feature representation of the U pieces of unlabeled data and the pseudo labels T Training is performed to obtain the encoder model Encoder of the T+1-th period T+1 and the model parameters θ thereof T+1 ; Step 3.
10. Update the model parameters θ for the T+1 period according to equation (11) T+1 T+1 ; 0' T+1 ←α·θ' r +(1-ω)θ T+1 (11) In formula (11), ω represents a momentum coefficient, and ω ∈ [0, 1); θ'T represents an updated model parameter of the Tth period, and when T = 0, let θ'T= θ T T ; Step 3.11, after assigning T + 1 to T, returning to step 3.1 for sequential execution.
2. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the Web short text data stream clustering method of claim 1, and the processor is configured to execute the program stored in the memory.
3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the Web short text data stream clustering method of claim 1.