Text data classification method
By employing a global-local semantic perception and prototype contrastive learning approach, we optimize text data classification, addressing the issues of long-tail distribution and dynamic evolution in social governance data. This enables efficient classification of text data and discovery of new categories, thereby enhancing the responsiveness of the social governance system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-26
- Publication Date
- 2026-03-10
AI Technical Summary
Existing text data classification methods struggle to effectively identify a small number of categories when faced with the long-tailed distribution and dynamic evolution of social governance data, resulting in poor classification reliability, inability to accurately identify new categories, and delayed response.
We employ a global-local semantic perception and prototype contrastive learning approach, optimizing semantic representation vectors through multi-head self-attention and dual-path attention models. By combining prototype clustering and hierarchical loss functions, we generate pseudo-labels for classification.
It improves the accuracy and reliability of text data classification, enabling it to accurately identify known categories and automatically discover new categories in long-tailed distributions and dynamic environments, thereby enhancing the adaptability and responsiveness of social governance systems.
Smart Images
Figure CN121636710A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and more specifically, to a method for classifying text data. Background Technology
[0002] When processing text data, it is necessary to classify the text data to improve the processing efficiency. For example, when processing social governance data, classifying the social governance data in a reasonable way ensures that the problems reflected by the social governance data are accurately located under the corresponding data categories, which greatly improves the efficiency of processing social governance issues based on social governance data.
[0003] However, due to the severe imbalance in data categories in social governance data, a small number of data categories have abundant sample data while a large number of data categories have scarce sample data. As a result, the current classification model is easily affected by the incomplete training data during training, and is prone to misclassification or omission, resulting in poor classification reliability. Summary of the Invention
[0004] This application provides a method for classifying text data to at least address the technical problem of poor classification reliability.
[0005] According to a first aspect of the embodiments of this application, a method for classifying text data is provided, the method comprising: Obtain the text data to be classified; The text data to be classified is semantically encoded to obtain a semantic representation vector, wherein the semantic representation vector is used to reflect the overall features of the text to be classified; The semantic representation vector is optimized globally and locally to obtain the text feature vector; Clustering is performed based on the similarity between the calculated text feature vectors and preset prototypes to obtain a cluster partitioning matrix, wherein the prototypes are representative feature vectors of each data category. The pseudo-labels of the text data to be classified are determined based on the text feature vector, the prototype, and the cluster partitioning matrix, wherein the pseudo-labels are used to characterize the probability that the text to be classified is assigned to each of the data categories; The text data to be classified is classified based on the pseudo-labels.
[0006] In conjunction with the first aspect, in an optional implementation of this application embodiment, the step of performing global and local optimization on the semantic representation vector to obtain the text feature vector includes: The text feature vector is obtained by globally optimizing the semantic representation vector using a multi-head self-attention model and locally optimizing the local features and feature sequence positions of the semantic representation vector using a dual-path attention model.
[0007] In conjunction with the first aspect, in an optional implementation of this application embodiment, the step of globally optimizing the semantic representation vector using a multi-head self-attention model includes: Weights are assigned to different words in the semantic representation vector based on their correlation.
[0008] In conjunction with the first aspect, in an optional implementation of this application embodiment, the step of using a dual-path attention model to locally optimize the local features and feature sequence positions of the semantic representation vector includes: The semantic representation vector is compressed, activated, and calibrated to obtain a first output value; The semantic representation vector is subjected to linear transformation, attention weight calculation, and weighted summation to obtain the second output value; The first output value and the second output value are fused to complete the local optimization.
[0009] In conjunction with the first aspect, in an optional implementation of this application embodiment, the step of compressing, activating, and calibrating the semantic representation vector to obtain a first output value includes: Compression: Through global average pooling, the semantic representation vectors are globally aggregated along the time dimension to obtain a global representation for each feature dimension. The calculation formula is as follows: ; in For the first A global semantic representation of each feature dimension. For time step In feature dimension The value, The sequence length; Incentives: The dependencies between feature dimensions are modeled using a two-layer fully connected network, and the weights of the feature dimensions are calculated using the following formula: ; in ReLU is a non-linear activation function. It is the sigmoid activation function. , , Weights for feature dimensions; Calibration: The calculated feature weights s are applied to the semantic representation vector embedding, and their calculation formula is as follows:
[0010] in, For the first Weights of each feature dimension, The time step in the semantic representation vector and feature dimensions The value, These are the calibrated feature values. That is, the first output value.
[0011] In conjunction with the first aspect, in an optional implementation of this application embodiment, the step of performing a linear transformation, attention weight calculation, and weighted summation on the semantic representation vector to obtain a second output value includes: A linear transformation is performed on the semantic representation vector, and a sequence-level feature representation is generated through a fully connected layer: ; in For learnable parameters, For the first Feature vectors at each time step; Along sequence dimension Apply the Softmax function to calculate the attention weights at each location: ; The position-aware sequence representation is obtained by weighted summation: ; in This represents element-wise multiplication. This is the second output value.
[0012] In conjunction with the first aspect, in an optional implementation of this application embodiment, before semantically encoding the text data to be classified to obtain a semantic representation vector, the method further includes: The text data to be classified is segmented and sequence transformed to obtain a word sequence; A global marker is added to the head of the word sequence, and the global marker is used to represent the overall semantic information; The step of semantically encoding the text data to be classified to obtain a semantic representation vector includes: The semantic representation vector is obtained by encoding the word sequence using a pre-trained language model.
[0013] In conjunction with the first aspect, in an optional implementation of this application embodiment, the step of clustering based on the similarity between the calculated text feature vector and a preset prototype to obtain a cluster partitioning matrix includes: The cosine similarity is obtained by multiplying the text feature vector with the prototype vector of the prototype. The text feature vectors and prototypes are grouped according to the cosine similarity to obtain prototype groups; The prototype group is divided into clusters to obtain the cluster division matrix.
[0014] In conjunction with the first aspect, in an optional implementation of this application embodiment, determining the pseudo-label of the text data to be classified based on the text feature vector, the prototype, and the cluster partitioning matrix includes: The pseudo-label is obtained by multiplying the text feature vector, the prototype vector of the prototype, and the cluster partitioning matrix.
[0015] In conjunction with the first aspect, in an optional implementation of the embodiments of this application, the method further includes: The prototype is optimized using the maximum entropy loss function, the accuracy of identifying the difference between the pseudo-label and the data category is optimized using the supervised clustering consistency loss function, and the pseudo-label is optimized using the group-level clustering consistency loss function. The maximum entropy loss function includes: ; in Assuming a priori uniform distribution The probability of evenly distributing the text data to be classified onto each prototype; The supervised clustering consistency loss function includes: ; in, This is represented by the pseudo-label distribution. The actual data category of the text data to be classified; The group-level clustering consistency loss function includes: ; in and This represents two pseudo-labels in a positive sample pair; The maximum entropy loss function, the supervised clustering consistency loss function, and the group-level clustering consistency loss function are weighted and combined as the overall optimization objective. The calculation formula is as follows: ; in , , These are the weight parameters.
[0016] This embodiment includes at least the following effects: In the process of classifying the text data to be classified, the semantic representation vector was optimized globally and locally. This ensured that the text feature vector could reflect both the overall and local characteristics of the text data to be classified. As a result, when calculating the cluster partitioning matrix, the cluster partitioning matrix integrated the global and local characteristics of the text data to be classified, which helped to improve the accuracy of pseudo-labels. This improved the classification accuracy and reliability of the text data to be classified. Attached Figure Description
[0017] Figure 1 This is a flowchart of a text data classification method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the system structure for applying the text data classification method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the processing procedure of the Local Part module provided in the embodiments of this application; Figure 4 This is a schematic diagram of the prototype cluster partitioning process provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the principle of prototype similarity calculation provided in the embodiments of this application. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0019] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.
[0020] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0021] In the process of building smart cities and modernizing social governance, data resources have become a core element for improving governance effectiveness. Relevant institutions and public service platforms urgently need to utilize efficient data analysis technologies to accurately perceive social dynamics and optimize resource allocation, thereby driving the transformation of governance models from passive response to proactive prediction. The massive amounts of public feedback information gathered through channels such as city service hotlines and social complaint platforms directly determine the scientific nature of social governance decisions, the rationality of public resource allocation, and the effectiveness of building a social trust system. Therefore, constructing an efficient, accurate, and dynamically adaptable large-scale social governance text data intelligent classification technology is crucial.
[0022] Current mainstream social governance text data classification technologies mainly rely on pre-defined classification rule systems and manual annotation mechanisms, combined with deep learning-based text classification models (such as BERT and GPT). However, these methods have significant limitations when dealing with complex data in real-world social governance scenarios, primarily in the following two aspects: Long-tail distribution and class imbalance: Social governance text data naturally exhibits severe class imbalance (long-tail distribution). High-frequency "head" issues (such as common complaints) have abundant samples, while a large number of low-frequency "tail" issues (i.e., "cold issues") have scarce samples. While existing deep learning methods (such as fine-tuned pre-trained models) have improved overall accuracy, when facing tail categories, the sparse training data makes it difficult for the model to learn effective feature representations, easily leading to low recognition rates or even missed identifications. Traditional strategies such as oversampling (which easily leads to tail overfitting), undersampling (which easily loses head information), or loss function adjustment (such as FocalLoss) are insufficient to effectively characterize the complex semantic features in social governance texts and cannot fundamentally solve the problem of weak tail category recognition. This results in an unbalanced or even neglected response of the governance system to low-frequency but potentially important social issues, leading to systemic risks such as distorted public opinion and risk accumulation.
[0023] The Challenges of Dynamic Evolution and New Category Identification: Social governance issues exhibit significant dynamic evolutionary characteristics, with new problem categories ("new problems") constantly emerging, such as public health emergencies and novel economic disputes. Existing classification methods (including mainstream deep learning models) typically assume a fixed set of categories (closed-world assumption), resulting in rigid classification boundaries and a lack of dynamic discovery and learning capabilities for new categories. When faced with unstructured, semantically novel new problem texts, traditional methods cannot effectively identify and classify them, often misclassifying them into known categories or labeling them as "other," leading to delayed or even absent governance responses. While some open-world learning or new category discovery methods exist (such as the ORCA framework), these methods typically assume a balanced distribution of data categories, failing to effectively address the long-tail distribution problem prevalent in social governance scenarios and highly overlapping with new problems. New categories (often stemming from low-frequency demands) are easily overwhelmed by the majority-class-dominated learning process, resulting in poor identification performance.
[0024] In summary, the core bottleneck faced by existing intelligent text classification technologies for social governance lies in: Insufficient ability to understand complex semantics and unstructured texts: It is difficult to effectively analyze the multi-granular semantic elements contained in social governance texts, such as the subject of the appeal, the type of problem, the spatial location, the implicit emotions and cross-domain connections, and it is easily affected by text noise and vague descriptions.
[0025] It cannot effectively overcome the recognition bias caused by the long-tail distribution: it has a weak ability to recognize tail categories (cold problems) with few samples, resulting in an unbalanced system response.
[0026] Lack of dynamic adaptation and category increment discovery capabilities: The classification system is rigid and cannot automatically and efficiently discover and identify emerging new problem categories (new classes) in an open environment, resulting in a response lag.
[0027] These limitations severely restrict the response speed, risk warning capabilities, and governance fairness of intelligent social governance systems. Therefore, there is an urgent need for a new intelligent classification method for social governance texts that can simultaneously and accurately parse complex semantics, effectively handle long-tail distributions, and dynamically discover new categories.
[0028] To address the aforementioned issues, this invention proposes a novel intelligent text classification technology that integrates global-local semantic perception with prototype contrastive learning. This technology enhances the identification of tail categories and the automatic discovery of new problem categories, thereby improving the adaptability, foresight, and fairness of the social governance system.
[0029] Based on this, this application provides a method for classifying text data. Taking the application of this text data classification method in the field of social governance as an example, refer to... Figure 1 The flowchart shown illustrates a text data classification method, which includes the following processing steps.
[0030] S100. Obtain the text data to be classified.
[0031] S102. Perform semantic encoding on the text data to be classified to obtain a semantic representation vector.
[0032] The semantic representation vector is used to reflect the overall features of the text to be classified.
[0033] S104. Perform global and local optimization on the semantic representation vector to obtain the text feature vector.
[0034] S106. Cluster the text feature vectors based on the similarity between the calculated text feature vectors and the preset prototypes to obtain a cluster partitioning matrix.
[0035] The prototype is a representative feature vector of each data category.
[0036] S108. Determine the pseudo-label of the text data to be classified based on the text feature vector, the prototype, and the cluster partitioning matrix.
[0037] The pseudo-labels are used to characterize the probability that the text to be classified is assigned to each of the data categories.
[0038] S110. Classify the text data to be classified according to the pseudo-labels.
[0039] In this embodiment, during the classification of text data, the semantic representation vector is optimized globally and locally. This ensures that the text feature vector reflects both the overall and local characteristics of the text data, allowing the cluster partitioning matrix to integrate both global and local features during subsequent calculations. This improves the accuracy of pseudo-labels, thereby enhancing the classification accuracy and reliability of the text data.
[0040] In one possible embodiment of this application, the step of performing global and local optimization on the semantic representation vector to obtain the text feature vector includes: The text feature vector is obtained by globally optimizing the semantic representation vector using a multi-head self-attention model and locally optimizing the local features and feature sequence positions of the semantic representation vector using a dual-path attention model.
[0041] Optionally, in one implementation of this embodiment, the step of globally optimizing the semantic representation vector using a multi-head self-attention model includes: Weights are assigned to different words in the semantic representation vector based on their correlation.
[0042] Optionally, in one implementation of this embodiment, the step of using a dual-path attention model to locally optimize the local features and feature sequence positions of the semantic representation vector includes: The semantic representation vector is compressed, activated, and calibrated to obtain a first output value; The semantic representation vector is subjected to linear transformation, attention weight calculation, and weighted summation to obtain the second output value; The first output value and the second output value are fused to complete the local optimization.
[0043] Optionally, in one implementation of this embodiment, the step of compressing, activating, and calibrating the semantic representation vector to obtain a first output value includes: Compression: Through global average pooling, the semantic representation vectors are globally aggregated along the time dimension to obtain a global representation for each feature dimension. The calculation formula is as follows: ; in For the first A global semantic representation of each feature dimension. For time step In feature dimension The value, The sequence length; Incentives: The dependencies between feature dimensions are modeled using a two-layer fully connected network, and the weights of the feature dimensions are calculated using the following formula: ; in ReLU is a non-linear activation function. It is the sigmoid activation function. , , Weights for feature dimensions; Calibration: The calculated feature weights s are applied to the semantic representation vector embedding, and their calculation formula is as follows: ; in, For the first Weights of each feature dimension, The time step in the semantic representation vector and feature dimensions The value, These are the calibrated feature values. That is, the first output value.
[0044] Optionally, in one implementation of this embodiment, the step of performing a linear transformation, attention weight calculation, and weighted summation on the semantic representation vector to obtain a second output value includes: A linear transformation is performed on the semantic representation vector, and a sequence-level feature representation is generated through a fully connected layer: ; in For learnable parameters, For the first Feature vectors at each time step; Along sequence dimension Apply the Softmax function to calculate the attention weights at each location: ; The position-aware sequence representation is obtained by weighted summation: ; in This represents element-wise multiplication. This is the second output value.
[0045] Optionally, in one implementation of this embodiment, before semantically encoding the text data to be classified to obtain a semantic representation vector, the method further includes: The text data to be classified is segmented and sequence transformed to obtain a word sequence; A global marker is added to the head of the word sequence, and the global marker is used to represent the overall semantic information; The step of semantically encoding the text data to be classified to obtain a semantic representation vector includes: The semantic representation vector is obtained by encoding the word sequence using a pre-trained language model.
[0046] Optionally, in one implementation of this embodiment, the step of clustering based on the similarity between the text feature vector and a preset prototype to obtain a cluster partitioning matrix includes: The cosine similarity is obtained by multiplying the text feature vector with the prototype vector of the prototype. The text feature vectors and prototypes are grouped according to the cosine similarity to obtain prototype groups; The prototype group is divided into clusters to obtain the cluster division matrix.
[0047] Optionally, in one implementation of this embodiment, determining the pseudo-label of the text data to be classified based on the text feature vector, the prototype, and the cluster partitioning matrix includes: The pseudo-label is obtained by multiplying the text feature vector, the prototype vector of the prototype, and the cluster partitioning matrix.
[0048] Optionally, in one implementation of this embodiment, the method further includes: The prototype is optimized using the maximum entropy loss function, the accuracy of identifying the difference between the pseudo-label and the data category is optimized using the supervised clustering consistency loss function, and the pseudo-label is optimized using the group-level clustering consistency loss function. The maximum entropy loss function includes: ; in Assuming a priori uniform distribution The probability of evenly distributing the text data to be classified onto each prototype; The supervised clustering consistency loss function includes: ; in, This is represented by the pseudo-label distribution. The actual data category of the text data to be classified; The group-level clustering consistency loss function includes: ; in and This represents two pseudo-labels in a positive sample pair; The maximum entropy loss function, the supervised clustering consistency loss function, and the group-level clustering consistency loss function are weighted and combined as the overall optimization objective. The calculation formula is as follows: ; in , , These are the weighting parameters.
[0049] In one specific implementation of this application embodiment, the text data classification method includes the following processing steps: like Figure 1 As shown, the method is through Figure 2 The system implementation shown specifically includes: It consists of two parts: a semantic feature encoding module and a comparative learning module based on prototype clustering.
[0050] The semantic feature encoding module consists of a semantic encoding layer and a feature extraction layer. The semantic encoding layer uses the pre-trained model BERT (Bidirectional Encoder Representations from Transformers) to transform social governance data text into deep semantic representation vectors, capturing the semantic information of the text. The feature extraction layer constructs an encoder based on the Global-Local-Attention mechanism. It enhances the modeling ability for fine-grained information through local feature extraction, while simultaneously combining textual context information to extract global features, thereby improving the understanding of long-tail data.
[0051] The prototype-based contrastive learning module learns from the feature-extracted data, utilizes prototypes to represent the distribution characteristics of the data, and generates a cluster partitioning matrix by calculating the similarity between prototypes and samples. In this process, the module can accurately identify known categories while effectively detecting potential new categories, thereby improving the model's generalization ability on long-tailed distribution data.
[0052] In one application scenario, Chinese-BERT-wwm is selected as the basic encoder, and the overall encoding process is as follows: Text Input and Word Segmentation: The original text is segmented into words to convert the input text into words. Convert to word sequence A special marker [CLS] is added at the beginning of the sequence to indicate overall semantic information.
[0053] BERT semantic vector extraction: Input the preprocessed text into the BERT model to obtain the vector representation of the text. ,in The vector representation of word i. As a global semantic vector of the entire text, it encompasses the comprehensive features of the text.
[0054] Vector optimization: To address the long-tail category problem in social governance scenarios, a global-local attention mechanism is further introduced to optimize the extracted semantic vectors, thereby enhancing the model's ability to identify low-frequency categories and improving the accuracy of text classification and topic discovery.
[0055] Using the methods described above, BERT can more precisely characterize text features in the task of category increment discovery for social governance text data, providing a solid semantic foundation for subsequent analysis and decision support.
[0056] (1) Feature optimization mechanism combining global-local attention BERT primarily relies on the Transformer structure for feature encoding, but its global self-attention mechanism may neglect important local features, affecting the recognition performance of long-tail categories. To address this, a Global-Local-Attention encoder is proposed, consisting of a multi-head self-attention (MSA) module and a local feature extraction (Local Part) module. MSA is responsible for learning long-distance dependencies, while the Local Part optimizes feature representation by modeling the dependencies between feature dimensions and sequence positions.
[0057] MSA Module: Global Information Modeling MSA learns the relationships between different words in a text through a multi-head self-attention mechanism and assigns them different weights. For example, in the text of a government hotline case, phrases like "the traffic light settings are unreasonable, and the green light time for left turns is too short" are crucial for case classification, while geographical information such as "xx Road in Wuchang District" contributes less. MSA captures the leap relationships of key information through the self-attention mechanism, reducing the interference of redundant features.
[0058] Local Part module: Local feature optimization Since MSA only focuses on global information and lacks fine-grained modeling of local features, this paper introduces the LocalPart module. This module uses Feature Dimension Attention (SE-Attention) and Sequence Position Attention (Seq-Attention) to weight the feature dimensions and time steps of the text embedding, respectively, to enhance the expressive power of key features.
[0059] (2) Dual-path attention mechanism for local feature enhancement like Figure 3 As shown, self-attention mechanisms have advantages in long-distance dependency modeling, but limitations in local feature interaction modeling, easily overlooking fine-grained relationships between adjacent words. To address this issue, this invention designs a dual-path Local Part module that simultaneously models feature dimensions and sequence positions, and superimposes their weights to achieve dynamic feature enhancement.
[0060] SE-Attention: SE-Attention primarily achieves dynamic weighting at the feature dimension level through three steps: Squeeze, Excitation, and Recalibration. The structure of SE-Attention is as follows: Figure 3 As shown.
[0061] 1) Squeeze (compression): Through global average pooling, text features are globally aggregated along the time dimension to obtain a global representation for each feature dimension. The calculation formula is as follows:
[0062] in For the first A global semantic representation of each feature dimension. Indicates time step In feature dimension The value, The sequence length is given.
[0063] 2) Excitation: This involves modeling the dependencies between feature dimensions using a two-layer fully connected network and calculating the weights of each feature dimension. The formula for calculating excitation is: , in This refers to the non-linear activation function ReLU. This represents the sigmoid activation function. , , These are the feature dimension weights, where each value represents the importance of a different feature dimension.
[0064] To limit computational complexity and enhance model generalization ability, a bottleneck structure is formed before the nonlinear transformation to parameterize the gating mechanism. This is achieved by creating a dimensionality reduction layer (with parameters) through two fully connected (FC) layers. and dimensionality reduction ratio A ReLU activation layer, followed by an up-dimensional layer (with parameters) ).
[0065] 3) Recalibration: The calculated feature weights s are applied to the original text embedding. The calculation formula is as follows:
[0066] in, Indicates the first Weights of each feature dimension, The time step in the original feature matrix and feature dimensions The value, This represents the calibrated feature value.
[0067] Seq-Attention: Time-Step Adaptive Modeling Since SE-Attention only weights based on the feature dimension and does not consider the specific contextual information at each position in the input sequence, which is crucial for processing sequential text, a Seq-Attention module was designed. This module first processes the original input... A linear transformation is performed, and a sequence-level feature representation is generated through a fully connected layer: (13) in For learnable parameters, For the first Feature vectors at each time step. Then along the sequence dimension. Apply the Softmax function to calculate the attention weights at each location: (14) Finally, the position-aware sequence representation is obtained by weighted summation: (15) in () indicates element-wise multiplication. This mechanism allows the model to focus on time steps that are important to the current task (such as keyword positions) while suppressing noise or redundant positions.
[0068] To retain the advantages of both feature dimension calibration and sequence position calibration, we fuse the output of SE-Attention with the output of Seq-Attention, so that key feature dimensions are further enhanced at important time steps, while unimportant feature dimensions are doubly suppressed at non-key positions.
[0069] Incremental findings in social governance data categories: To address the long-tail distribution problem in social governance data, a contrastive learning method based on prototype clustering is proposed to simultaneously identify known categories and discover new categories. This method introduces a set of dynamically updated prototype vectors and, guided by a maximum entropy loss function, effectively learns a representative sample distribution in the feature space, alleviating the difficulty in identifying categories with small sample sizes.
[0070] The model generates a cluster partitioning matrix by calculating the similarity between samples and prototypes, and then uses this matrix to generate pseudo-labels. An introduced clustering consistency loss function compares the pseudo-labels and real labels under different views, further improving the model's stability and ability to discover new categories.
[0071] The overall approach consists of three parts: prototype clustering, pseudo-label generation, and hierarchical loss optimization. It is suitable for the task of discovering category increments in social governance data.
[0072] (1) Prototype clustering-driven incremental category discovery To address the challenge of identifying long-tail categories in social governance data, a prototype clustering mechanism is introduced to progressively identify and absorb new categories, thus discovering category increments. The core of this mechanism lies in the fact that as the training process progresses, prototypes continuously update and aggregate new representational features, gradually covering samples that were not clearly categorized in the early stages of training. When certain samples form new clustering patterns in the feature space, the system can classify them into new prototype clusters. Through the semantic organization capabilities between prototypes, unknown categories can be effectively modeled and identified, achieving continuous learning and discovery of long-tail categories.
[0073] The purpose of prototype clustering is to generate a cluster partitioning matrix of prototypes, which is used for subsequent pseudo-label generation. Data features and prototype All have undergone After standardization, the product of the feature vector and the prototype vector is the cosine similarity between them, calculated using the following formula: , From a prototype perspective, each column in q represents the similarity of a specific prototype to features from other samples; the greater the similarity, the more representative the prototype is of the sample. Intuitively, if two prototypes share more common representation instances, they are more likely to belong to the same category. Therefore, we calculate the similarity score between two prototypes by calculating the Jaccard distance between the sample sets they represent, using the following formula:
[0074] in Representative prototype The set of all representative instances. In each epoch, an affinity matrix A is formed by calculating the similarity between every two prototypes. This matrix is used for spectral clustering between prototypes, resulting in a cluster partitioning matrix G. Therefore, matrix A is the adjacency matrix in the spectral clustering process to detect densely connected prototypes, which are considered prototype groups. More reliable prototype groups can be gradually obtained through feature extractors and enhancements to the representational power of prototypes. The process of clustering samples by prototypes to generate a cluster partitioning matrix is as follows: Figure 4 As shown.
[0075] (2) Pseudo-tag generation and optimization The generation process of pseudo-tags can be described by a simple matrix multiplication, and the calculation formula is as follows: , in It is the feature vector generated after the sample passes through the feature layer. It is a set of randomly initialized prototypes, whose dimensions are... Having the same dimensions, Each value in the equation is considered as the cosine similarity between the sample and the prototype. The cluster partitioning matrix describes the grouping of the prototypes. This indicates that the i-th prototype is contained within the j-th cluster, and the matrix represents this. It is generated by an independent prototype clustering process. It is the final calculated pseudo-label of the sample category, and the pseudo-labels of both labeled and unlabeled samples can be calculated in this unified way. The pseudo-label can be regarded as the probability of the sample being assigned to the prototype.
[0076] (3) Design of hierarchical loss function To effectively improve the performance of incremental category discovery, especially the ability to identify few categories in long-tailed distribution scenarios, a hierarchical loss function consisting of three parts was designed: prototype layer loss, supervised clustering consistency loss, and group-level consistency loss, which together constitute the final optimization objective.
[0077] Maximum entropy loss of prototype layer: Prototypes, as compact representations of a class of samples in the feature space, aim to capture the main feature information of the class. Traditional clustering methods (such as k-means) are usually based on ideal assumptions such as single center per class and Gaussian distribution, which limit their effectiveness in asymmetric and non-convex data distributions. However, by introducing multiple prototypes, the complex structure of each class can be more flexibly characterized, especially suitable for long-tailed classes.
[0078] To ensure that each prototype effectively carries representative information, maximum entropy loss is introduced. This encourages the model to distribute samples evenly across different prototypes, avoiding the "collapse" phenomenon where all samples are concentrated on a few prototypes. The formula for calculating this loss is:
[0079] in Assuming a priori uniform distribution This loss term, which assigns the probability of samples to each prototype on an even basis, is particularly crucial in dealing with class imbalance caused by long-tail distributions, and can effectively improve the attention and learning effect of tail classes.
[0080] Supervised clustering consistency loss: Since only about 5% of the samples in the citizen service hotline and city message board data have manual labels, after aligning the pseudo-labels with the real labels, cross-entropy loss is used to measure the difference between them, thereby improving the discrimination ability of labeled samples. The specific calculation method is shown in equation (20): (20) in, This is represented as the distribution of pseudo-labels predicted by the model. The true label of the sample. This loss term helps the model learn clear class boundaries even with scarce supervised information.
[0081] Group-level clustering consistency loss: To learn better representations to optimize prototype clustering results, the pseudo-labels generated by the model need to be validated to match the true class hierarchy. Therefore, a group-level clustering consistency loss is introduced to determine whether a trained representation and cluster partitioning are reasonable and to ensure that the augmented pseudo-labels for the same sample remain as consistent as possible. Its calculation formula is as follows:
[0082] in and The pseudo-labels represent the two instances in a positive sample pair. They can be regarded as group-level pseudo-labels for each other, so that the model can minimize the difference in pseudo-labels between the same sample.
[0083] The three loss functions mentioned above are weighted and combined to form the overall optimization objective of the model. The calculation formula is as follows: , in , , The weight parameter determines the importance of the loss of each component and can be dynamically adjusted during training to adapt to the learning objectives at different stages.
[0084] In summary, by integrating semantic enhancement and prototype-driven two-stage technical frameworks, the parsing capability of multi-granular semantic features in social governance data can be effectively improved, and balanced processing of long-tailed distributed data can be achieved. This technology can not only accurately identify known categories of text, but also automatically discover new categories in open environments, providing continuously optimized classification system support for intelligent social governance.
[0085] The sequence numbers or order of description of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0086] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method of classifying text data, characterized by, The method comprises: acquiring text data to be classified; performing semantic coding on the text data to be classified to obtain a semantic representation vector, wherein the semantic representation vector is used to reflect overall features of the text to be classified; performing global optimization and local optimization on the semantic representation vector to obtain a text feature vector; performing clustering according to similarity between the text feature vector and a preset prototype to obtain a cluster partition matrix, wherein the prototype is a representative feature vector of each data category; determining a pseudo label of the text data to be classified according to the text feature vector, the prototype and the cluster partition matrix, wherein the pseudo label is used to represent a probability of the text to be classified being divided into each data category; classifying the text data to be classified according to the pseudo label.
2. The method of classifying text data according to claim 1, wherein, The global optimization and local optimization of the semantic representation vector to obtain a text feature vector comprises: performing global optimization on the semantic representation vector by using a multi-head self-attention model and performing local optimization on local features and feature sequence positions of the semantic representation vector by using a double-path attention model to obtain the text feature vector.
3. The method of classifying text data according to claim 2, wherein, The global optimization of the semantic representation vector by using a multi-head self-attention model comprises: assigning corresponding weights according to relevance of different words in the semantic representation vector.
4. The method of claim 2, wherein, The local optimization of the local features and feature sequence positions of the semantic representation vector by using a double-path attention model comprises: performing compression, excitation and calibration on the semantic representation vector to obtain a first output value; performing linear transformation, attention weight calculation and weighted summation on the semantic representation vector to obtain a second output value; fusing the first output value and the second output value to complete the local optimization.
5. The method of classifying text data according to claim 4, wherein, The compression, excitation and calibration of the semantic representation vector to obtain a first output value comprises: compression: performing global aggregation on the semantic representation vector along the time dimension by global average pooling to obtain a global representation of each feature dimension, and the calculation formula is: ； wherein is a global semantic representation of the th feature dimension, is a time step, is a value of the feature dimension at the th sequence length; excitation: modeling the dependency relationship between feature dimensions by two fully connected networks to calculate the weight of the feature dimension, and the calculation formula is: ； wherein is a non-linear activation function ReLU, is a sigmoid activation function, , , is a feature dimension weight; calibration: the calculated feature weight s acts on the semantic representation vector embedding, and the calculation formula is: ； wherein, is a weight for the th feature dimension, is a value of the th feature dimension at the th time step in the semantic representation vector, is a calibrated feature value, i.e., the first output value.
6. The method of classifying text data according to claim 4, wherein, The linear transformation, attention weight calculation and weighted summation of the semantic representation vector to obtain a second output value comprises: performing linear transformation on the semantic representation vector to generate sequence-level feature representation by a fully connected layer: ； wherein is a learnable parameter, is a feature vector at the th time step; Along the sequence dimension Apply the Softmax function to compute the attention weights for each position: ； obtaining position-aware sequence representation by weighted summation: ； wherein represents an element-wise multiplication, is a second output value.
7. The method of classifying text data according to claim 1, wherein, Before the semantic coding of the text data to be classified to obtain a semantic representation vector, the method further comprises: performing word segmentation processing on the text data to be classified and performing sequence conversion to obtain a word sequence; adding a global mark at the head of the word sequence, and the global mark is used to represent overall semantic information; The semantic coding of the text data to be classified to obtain a semantic representation vector comprises: performing coding processing on the word sequence by using a pre-trained language model to obtain the semantic representation vector.
8. The method of classifying text data according to claim 1, wherein, The clustering according to the similarity of the text feature vector and a preset prototype includes: Obtaining a cosine similarity by using a product of the text feature vector and a prototype vector of the prototype; Grouping the text feature vector and the prototype according to the cosine similarity to obtain a prototype group; Performing cluster partitioning on the prototype group to obtain the cluster partitioning matrix.
9. The method of claim 1, wherein, The determining of the pseudo label of the text data to be classified according to the text feature vector, the prototype and the cluster partitioning matrix includes: Multiplying the text feature vector, a prototype vector of the prototype and the cluster partitioning matrix to obtain the pseudo label.
10. The method of classifying text data according to claim 1, wherein, The method further includes: Optimizing the prototype by using a maximum entropy loss function, optimizing the difference identification accuracy of the pseudo label and the data category by using a supervised cluster consistency loss function, and optimizing the pseudo label by using a group-level cluster consistency loss function; The maximum entropy loss function includes: ； wherein is a uniform distribution a priori, is the probability that the text data to be classified is equally distributed over each prototype; The supervised cluster consistency loss function includes: ； wherein, is represented as the pseudo label distribution, is the actual data class of the text data to be classified; The group-level cluster consistency loss function includes: ； wherein and denote the two pseudo-labels in a positive sample pair; Combining the maximum entropy loss function, the supervised cluster consistency loss function and the group-level cluster consistency loss function as a whole optimization target by weighting, and a calculation formula is as follows: ； wherein , , are weight parameters.