Online semi-supervised classification algorithm based on multi-label evolving high-dimensional text stream

Abstract

The application provides an online semi-supervised classification algorithm based on multi-label evolution high-dimensional text flow, characterized by comprising model initialization, classification stage and model maintenance; D init labelled documents are taken, and Z min micro-clusters are created for each label, and the micro-clusters contain documents; an initial model makes a prediction for each incoming document flow, for each incoming document, the model calculates cluster-document probability, and based on the probability score, selects k nearest micro-clusters Zd; at this time, the number of labels Y to be predicted is equal to the number of micro-clusters with a higher than Zd distribution mean l count ; when the number of clusters greater than the average is one, the label with a high cluster number in the nearest cluster is predicted, otherwise the sum of cluster probabilities of each label is compared, and the label co-occurrence score of the nearest label is used for prediction; after the label is predicted, if the incoming document has no label, it is added to the nearest micro-cluster of each predicted label, otherwise the incoming document is added to the nearest micro-cluster of the ground truth label.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an online semi-supervised classification algorithm based on multi-label evolution of high-dimensional text flow. BACKGROUND

[0002] The goal of multi-label learning model is to predict all labels corresponding to each input instance. When processing multi-label text flow, there are many challenges.

[0003] For example, high dimensionality. As the number of features in text data increases, the dimensionality of the problem also increases, which leads to difficulty in feature selection, increased computational complexity, and dimensionality curse. In high dimensionality, feature selection becomes more difficult because there are a large number of features, many of which can be redundant or irrelevant. Selecting an appropriate feature subset is crucial for improving classification performance and reducing computational complexity. High-dimensional data increases the complexity of computation. In high dimensionality, operations such as calculating distance, similarity, and probability become more time-consuming, leading to a decrease in algorithm efficiency. Dimensionality curse refers to the fact that in high-dimensional space, sample density becomes very sparse, leading to a decrease in the performance of classification algorithms.

[0004] For example, label scarcity. It is difficult to accurately learn and predict rare labels in the absence of sufficient instances. Due to the lack of label instances, the model cannot capture the features and patterns of these labels, resulting in a decrease in prediction performance.

[0005] Therefore, high dimensionality, label scarcity, and other issues are important challenges when processing multi-label text flow. Appropriate methods need to be adopted to address issues such as feature selection, computational complexity, and dimensionality curse, improve learning and prediction ability for rare labels, and improve the performance and efficiency of the algorithm. SUMMARY

[0006] To address the deficiencies in the prior art, the present application provides an online semi-supervised classification algorithm based on multi-label evolution of high-dimensional text flow, which solves the problem of difficulty in feature selection, increased computational complexity, and dimensionality curse caused by the increase in the number of features in text data as the dimensionality of the problem increases.

[0007] The above technical purpose of the present application is achieved by the following technical solution:

[0008] An online semi-supervised classification algorithm based on multi-label evolution of high-dimensional text flow includes model initialization, classification stage, and model maintenance.

[0009] Take D init labelled documents, and create Z minA micro-cluster, the micro-cluster contains documents; the initial model makes prediction for each incoming document stream, for each incoming document, the model calculates the cluster-document probability, and based on the probability score, selects the k nearest micro-cluster Zd; at this time, the number of labels Y that need to be predicted is equal to the number of micro-clusters with a higher number than the average of Zd distribution l count ; when the number of clusters greater than the average is one, the label with the highest cluster number in the nearest cluster is predicted, otherwise the sum of the cluster probability of each label is compared, and the label co-occurrence score of the nearest label is used for prediction; after predicting the label, if the incoming document has no label, it is added to the nearest micro-cluster of each predicted label, otherwise the incoming document is added to the nearest micro-cluster of the ground truth label, and the co-term score V between each mispredicted label related cluster and the document is reduced d∩z .

[0010] The application further provides that: the micro-cluster is defined as an 8-tuple, Where m z is the number of documents, represents the word frequency of word w, n z is the sum of the frequency count of all words in the cluster Lz stores the assigned label of the micro-cluster, rz contains the decay weight, u z is the last updated timestamp, ta z is the timestamp of the arrival of the word, cw z is the co-occurrence score matrix of the word and the word, each entry is defined as follows,

[0011]

[0012] Here, is the frequency count of word w in document d', w i and w j between them must satisfy Where i≠j.

[0013] The application further provides that: when the model is initialized, D init labeled instances are used, for each label l∈L, a set of equal size instances S init is selected from the given D i , defined as S init ={S1,…,S |L|}, the latent subspace of the text data is processed using LDA, and the first-level label co-occurrence weight matrix LC M is calculated, where each entry uses heuristic probability to calculate the weight between labels l i and l j , defined as

[0014]

[0015] The application is further configured to: after initializing the cluster and label co-occurrence score matrix, perform a classification process for each arriving document, including two steps: calculating the similarity score of each arriving instance in the stream with all active clusters in the model, and predicting the label by observing the k nearest clusters.

[0016] The application is further configured that in the similarity calculation, the probability score is defined as:

[0017]

[0018] where D is the total number of active documents in the model M, ICF w is the inverse cluster frequency, which is important for calculating the weight, defined as

[0019] The application is further configured that model maintenance includes concept evolution, deletion of outdated label-related terms, and merging of micro-clusters.

[0020] The application has the following beneficial effects:

[0021] 1. Efficient processing of multi-label text stream: The online semi-supervised classification algorithm proposed in the application can process multi-label text stream data in real time. It can dynamically adapt to concept drift and label sparsity in data stream, and has low computational and memory requirements. This enables the algorithm to run efficiently in large-scale data streams and real-time applications.

[0022] 2. Model adaptive ability: The algorithm in the application has strong adaptive ability, which can handle concept drift and changes in label cardinality. It can adjust the model in time, capture new concepts and label associations in the data stream, and accurately predict new instances. This enables the algorithm to maintain accuracy and performance in a dynamic environment.

[0023] 3. The algorithm in the application considers label correlation when predicting labels. By embedding label co-occurrence probability and clustering similarity, the algorithm can capture the association between labels and improve prediction performance. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The overall flowchart of the online semi-supervised classification algorithm proposed in the application. DETAILED DESCRIPTION

[0025] The technical solutions in the application will be further described below in conjunction with the drawings and examples.

[0026] In this application, an online semi-supervised classification algorithm based on multi-label evolving high-dimensional text streams is proposed. By exploiting a small number of labeled instances, the algorithm dynamically maintains a term subspace for each label and uses a set of evolving micro-clusters. For multi-label classification, the algorithm uses a non-parametric Dirichlet model to predict the k nearest micro-clusters. To handle gradual concept drift in the term space, a triangular time function is used to compute the difference between the term arrival time and the cluster lifetime. For sudden concept drift, the algorithm uses two steps: (a) remove outdated micro-clusters by exploiting an exponential decay function, (b) create new micro-clusters based on the Dirichlet process using a Chinese restaurant process.

[0027] The online semi-supervised classification algorithm model consists of three phases, model initialization, classification phase, and model maintenance. A small portion of the document stream is assumed to be labeled while the majority of the document stream is unlabeled. For example, Figure 1 The overall flowchart of the algorithm model of this application is shown. First, a small number of labeled documents (D init ) are taken and a small number of micro-clusters (Z min ) are created for each label. The micro-clusters contain the documents. For each incoming document, the model will compute the cluster-document probabilities and then based on the probability scores, the k nearest micro-clusters (Zd) are selected. At this stage, the number of labels to be predicted (Y) is equal to the number of micro-clusters with a higher than the mean of Zd distribution (l count ). If the number of clusters with a higher than the mean is one, the label is predicted in the nearest cluster with a high cluster number. Otherwise, the sum of the cluster probabilities for each label is compared and the label co-occurrence score of the nearest label is considered to make the prediction. After the label is predicted, if the incoming document is unlabeled, it is added to the nearest micro-cluster of each predicted label. Otherwise, the incoming document is added to the nearest micro-cluster of the ground-truth label and the common term score (V d∩z ) between each mispredicted label related cluster and the document will be slightly reduced.

[0028] Regarding micro-clusters:

[0029] A micro-cluster (in the stream) is represented by a cluster feature set that contains different statistics about the instances in it. The feature set of a micro-cluster z is defined as an 8-tuple Here, m z is the number of documents, is the term frequency of a word w, n z is the sum of the frequency counts of all words in the cluster Lz stores the assigned label of the micro-cluster, rz contains the decay weight, u z is the last update timestamp, ta z is the word arrival timestamp. cw zis the word-to-word co-occurrence score matrix, each entry is defined as follows,

[0030]

[0031] Here, is the frequency count of word w in document d'. In document d, w i is the ratio between w j where i≠j.

[0032] Document d is added to cluster z through the addable attribute of micro-cluster.

[0033]

[0034] Here, N d is the total number of words in document d, is the frequency count of each word in the document, cw d denotes the co-occurrence matrix of the document, ta d combines the arrival time of the terms associated with the document, N d is the length of the document. The update timestamp u z of the micro-cluster will be the current timestamp of the model. The complexity of adding a document to the update cluster is where is the average length of the document.

[0035] The initialization of the model starts with the initialization of the learning model that takes D init labeled instances. First, for each label l e L, a set of equal size instances S init is selected from the given D i . Formally, define S init = {S1,..., S |L|}, where S i may be overlapping or non-overlapping with each other due to the presence of multi-label data. The instances of each label S i are then partitioned into Z min partitions (clusters) that effectively capture the latent subspaces of the text data using LDA. The primary label co-occurrence weight matrix LC M is computed, where each entry computes the weight between labels l i and l j using a heuristic probability defined as

[0036]

[0037] ​After initializing the cluster and label co-occurrence score matrix, we start the classification of each arriving document. The classification phase consists of two steps: (1) computing the similarity score of each arriving instance in the stream with all active clusters in the model, and (2) predicting the labels by observing the k nearest clusters.

[0038] Similarity computation: In the past decade, a series of algorithms proposed distance-based or probability-based similarity to predict the labels. The former type of measure suffers from the curse of dimensionality, and in the latter approach, designing a probability-based similarity is still a challenging task. Most of the probability-based approaches deal with short texts (instances have only low-dimensional representation), or long texts (instances have only high-dimensional representation). Therefore, to deal with multi-label documents containing short and long texts, a novel probability score is defined as follows:

[0039]

[0040] D is the total number of active documents in the model M. ICF w is the inverse cluster frequency, which is important to compute the weight, defined as

[0041]

[0042] Predicting labels: After computing the similarity between the arriving document and all active clusters of the model using Equation 3, to predict the labels, the k nearest clusters with high probability scores are selected. The number of predicted labels will be the number of clusters that are higher than the average in the normalized probability distribution of the k nearest clusters. The label co-occurrence probability, i.e., the conditional probability in Equation 2, has been computed in the initialization process. The labels are predicted by combining the frequent pairs of nearest labels and their cluster counts. After predicting the labels, the document is added to the selected micro-cluster in the process of selection (in Equation 2) and creation of new clusters (in Equation 4).

[0043] Model maintenance includes the conceptual evolution of the non-parametric model, the deletion of outdated label-related terms, and the merging of micro-clusters.

[0044] Conceptual evolution of the non-parametric model: Each active micro-cluster contains different term spaces with different distributions. Therefore, the basic idea is to check whether the new document belongs to the nearest term space of the existing active micro-clusters (k nearest micro-clusters) or to a new term space. Using Equation 3, the probability with the existing micro-clusters is computed. The probability of creating a new micro-cluster is defined as

[0045]

[0046] At this point, the popularity of the clusters can be seen, where a is a small pseudo-value and β is a small pseudo-occurrence number of the terms of the new micro-cluster. Therefore, the condition for selecting a micro-cluster for the arriving document is

[0047]

[0048] Deleting outdated label-related terms: In addition to creating microclusters for current concepts, the model also needs to consider deleting outdated concepts. A method using a forgetting mechanism to delete outdated microclusters is adopted. In the decay mechanism, if a microcluster is not updated, its importance weight (r) is reduced. z The value will decrease over time. If the microcluster is updated over time, this means it can capture the current concept. For simplicity, the definition of an exponential decay function is used, as shown below:

[0049]

[0050] t M This is the current time of the model. r is set whenever a document is added to the micro-cluster z. z =1. If a microcluster does not receive documents for a period of time, its score will approach zero, leading to the deletion of the microcluster.

[0051] The core terms related to the labels in each term space may or may not be mutually exclusive; however, the importance of these label-related terms can change over time in the text stream. To address this issue, a cluster feature ta is maintained, storing the arrival time series of terms in each micro-cluster. z Based on trigonometric time, the age of the cluster is calculated, and a recentity score is calculated accordingly for each term. If a term's recentity score is less than a threshold Γ defined by us, then that term is not considered a representation of the cluster's active concept. A trigonometric time decay function is used to fade outdated clusters in the model, defined as follows:

[0052] Δf(T)=((T 2 +T) / 2).

[0053] Here, T is the timestamp number. The term's recency score... w It is measured by the ratio of the sum of the number of arrivals and the age of the cluster. For example, if m z =9, then the age of cluster z will be:

[0054] Age z =Δf(9)-Δf(1)=((9) 2 +9) / 2)-((1 2 +1) / 2)

[0055] Assuming the word w i It appears in three documents in clusters D1, D2, and D8, with arrival times of 1, 2, and 8, respectively. The word w i Arrival time in the cluster is stored as: wi ={timestamp:1,timestamp:2,timestamp:8}

[0056] And w i The recency score is calculated as follows:

[0057] Analysis reveals that as new documents arrive in the micro-cluster, more and more terms become unimportant. In this way, over time, terms are gradually and automatically filtered out.

[0058] Merging Microclusters: The multinomial distribution parameter β is responsible for calculating the homogeneity between documents and microclusters. However, as the number of noisy terms in a cluster increases over time, the probability of noisy terms may dominate and overwhelm that of core terms. After conceptual drift of tag-related terms and gradual removal of noisy and outdated terms, two microclusters may share a highly overlapping term space, thus affecting the granularity of the clusters. In this application, Equation 3 is used to incorporate the cluster merging process into the calculation of the probability between two clusters. Over time, if a cluster is not updated, then r z The value will be close to zero, indicating that the cluster is outdated. Therefore, before deleting a cluster, the probability is calculated using each active cluster with the same label, while simultaneously calculating the probability that the cluster selects a new cluster. If the probability of the new cluster is greater than the probability of the active cluster, delete the micro-cluster. Otherwise, merge the micro-cluster with the nearest micro-cluster containing the same label.

[0059] This invention has many advantages and positive effects compared with the prior art. It can effectively process multi-labeled text streams, adapt to dynamic environmental changes, and improve prediction performance and efficiency.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An online semi-supervised classification algorithm based on multi-label evolutionary high-dimensional text streams, characterized by: This includes model initialization, classification phase, and model maintenance; Take D init A document with tags, and create a Z tag for each tag. min There are three micro-clusters, each containing a document. The initial model predicts for each incoming document stream. For each incoming document, the model calculates the cluster-document probability and selects the k nearest micro-clusters Zd based on the probability scores. At this point, the number of labels Y to be predicted is equal to the number l of micro-clusters l that have a distribution mean higher than Zd. count When the number of clusters greater than the average is one, a label with a high number of clusters in the nearest cluster is predicted; otherwise, the sum of the cluster probabilities for each label is compared, and the label co-occurrence score of the nearest label is used for prediction. After predicting the label, if the arriving document has no label, it is added to the nearest micro-cluster of each predicted label; otherwise, the arriving document is added to the nearest micro-cluster of the ground truth label, reducing the common term score V between each mispredicted label-related cluster and document. d∩z ; Microclusters are defined as 8-tuples. Where m z It is the number of documents. n represents the word frequency of word w. z It is the sum of the frequency counts of all words in the cluster. L z The allocation tags of the microclusters are stored, r z Includes decay weights, u z It is the last update timestamp, ta z It's the timestamp of the word's arrival, cw z This is a co-occurrence score matrix of words, with each entry defined as follows: here, It is the frequency count of word w in document d′, where w is the most frequent word in the document. i and w j The ratio between them must meet the following conditions. Where i≠j; During model initialization, D is used init For each labeled instance, l∈L, from a given D init Select a set of instances S of equal size. i Defined as S init ={S1,…,S |L| LDA is used to process the latent subspace of the text data, and the first-level label co-occurrence weight matrix LC is calculated. M Each entry uses a heuristic probability calculation to determine the label l. i and l j The weights between them are defined as follows: After initializing the cluster and label co-occurrence score matrix, a classification process is performed for each arriving document, which includes two steps: calculating the similarity score between each arriving instance in the stream and all active clusters in the model, and predicting the label by observing the k nearest clusters; When calculating similarity, the probability score is defined as: Where D is the total number of active documents in model M, and ICF w It is the inverse cluster frequency, used to calculate the importance of weights, defined as...

2. The online semi-supervised classification algorithm based on multi-label evolutionary high-dimensional text stream as described in claim 1, characterized in that: Model maintenance includes concept evolution, removal of outdated label-related terms, and merging of microclusters.

Images