Label propagation text classification method and device generated by fusing weak supervision information

Abstract

The invention discloses a label propagation text classification method and device generated by fusing weak supervision information, and relates to the technical field of text classification. The method comprises the steps of selecting an initial category word set, pre-training an initial multi-label text classification model, inputting an original text into an encoder layer to obtain deep potential features, and inputting the deep potential features into a prediction layer to obtain an initial classification prediction result; determining a pseudo label set; gradually updating the pseudo label set, and further training the pre-trained multi-label text classification model; an integrated pseudo label set is obtained; determining an adjacent matrix of the k-neighbor graph; determining a label correlation matrix; performing noise correction on the integrated pseudo label set, and performing final training on the multi-label text classification model; and according to the trained multi-label text classification model, obtaining a label corresponding to the to-be-classified text. According to the invention, the noise supervision information is corrected by using the text neighbor relation and the label correlation, so that the classification accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of text classification technology, and in particular to a method and apparatus for text classification based on label propagation generated by integrating weakly supervised information. Background Technology

[0002] Multi-label text classification refers to the task of learning a classification model from texts that simultaneously correspond to multiple category labels. Generally speaking, multi-label text classification is a fundamental task in text information management within natural language processing, because in practical applications, texts with rich semantics often belong to multiple category labels simultaneously. For example, an academic paper may cover multiple disciplines and research directions, and a news report may involve multiple topics. Over the past few decades, various supervised learning methods have emerged in the field of multi-label text classification research, used to automatically assign a set of relevant labels to new texts. Typically, the learning process of these multi-label text classification methods includes two steps: (1) constructing a manually labeled training text dataset; and (2) learning a classification model on this training dataset. In the first step, the annotator needs to first define the set of category labels and then manually assign each training text to its associated label. However, accurately labeling training texts for multi-label tasks is not only time-consuming but also more costly than labeling for multi-class classification tasks.

[0003] To reduce the labor cost of manual annotation, a feasible solution is to collect lower-cost weakly supervised annotation information and then train a classification model using weakly supervised learning techniques. Weakly supervised text classification methods can include both multi-class and multi-label approaches. Currently, there are relatively few weakly supervised learning methods specifically for multi-label text classification (MLTC). For multi-class scenarios, early topic-based modeling methods constructed topic-specific priors by counting the occurrences of category words, while recent research focuses more on refining weakly supervised information through various techniques, such as self-training. Compared to multi-class tasks, weakly supervised learning faces several additional challenges in multi-label text classification. First, category relevance means that some words may belong to multiple categories simultaneously, making the selection of the most representative category words more complex. For example, in the multi-label dataset AAPD, the categories "Machine Learning" and "Artificial Intelligence" often appear simultaneously in the same text, and representative words such as "supervision," "learning," and "neural" may correspond to both, thus creating semantic ambiguity when manually selecting category words. Second, in multi-label scenarios, the supervision signal generated by category words has greater noise. Such high-noise supervision information makes it difficult to directly learn a reliable classification model, resulting in low classification accuracy. Summary of the Invention

[0004] To address the low classification accuracy problem in existing technologies, this invention provides a method and apparatus for classifying label-propagating text by fusing weakly supervised information. The technical solution is as follows:

[0005] On the one hand, a label propagation text classification method that integrates weakly supervised information is provided. This method is implemented by a label propagation text classification device that integrates weakly supervised information. The method includes: S1. Select an initial set of category words, and pre-train the initial multi-label text classification model based on the initial set of category words to obtain a pre-trained multi-label text classification model, which includes an encoder layer and a prediction layer. S2. Input the original text into the encoder layer to obtain deep latent features, and input the deep latent features into the prediction layer to obtain the initial classification prediction result; S3. Determine the set of pseudo-tags based on the frequency of category words in the original text; S4. The pseudo-label set is updated step by step using the pseudo-label set and sharpened classification prediction. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained. S5. Obtain the integrated pseudo-label set based on the further trained multi-label text classification model; S6. Determine the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder. S7. Determine the tag correlation matrix based on the parameter matrix of the encoder layer; S8. Based on the adjacency matrix and the label correlation matrix, noise correction is performed on the integrated pseudo-label set. Based on the noise-corrected pseudo-label set, the multi-label text classification model is trained for the final training to obtain the trained multi-label text classification model. S9. Input the text to be classified into the trained multi-label text classification model to obtain the label corresponding to the text to be classified.

[0006] On the other hand, a label propagation text classification device that integrates weakly supervised information is provided. This device is applied to the label propagation text classification method that integrates weakly supervised information. The device includes: The pre-training module is used to select an initial set of category words and pre-train the initial multi-label text classification model based on the initial set of category words to obtain a pre-trained multi-label text classification model, which includes an encoder layer and a prediction layer. The feature extraction module is used to input the original text into the encoder layer to obtain deep latent features, and then input the deep latent features into the prediction layer to obtain the initial classification prediction results. The statistics module is used to determine the set of pseudo-tags based on the frequency of category words in the original text; The update module is used to progressively update the pseudo-label set using the pseudo-label set and sharpened classification predictions. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained. The integration module is used to obtain an integrated set of pseudo-labels based on the further trained multi-label text classification model. The first determining module is used to determine the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder. The second determining module is used to determine the tag correlation matrix based on the parameter matrix of the encoder layer; The training module is used to correct the noise of the integrated pseudo-label set based on the adjacency matrix and the label correlation matrix, and to finally train the multi-label text classification model based on the noise-corrected pseudo-label set to obtain the trained multi-label text classification model. The classification module is used to input the text to be classified into a trained multi-label text classification model to obtain the label corresponding to the text to be classified.

[0007] On the other hand, a label propagation text classification device that integrates weakly supervised information is provided. The label propagation text classification device that integrates weakly supervised information includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, they implement any of the methods described above for label propagation text classification that integrates weakly supervised information.

[0008] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described label propagation text classification methods that integrate weakly supervised information.

[0009] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: An initial set of category words is selected, and an initial multi-label text classification model is pre-trained based on this set. This pre-trained model includes an encoder layer and a prediction layer. The original text is input into the encoder layer to obtain deep latent features, which are then input into the prediction layer to obtain initial classification prediction results. A pseudo-label set is determined based on the frequency of category words in the original text. The pseudo-label set is progressively updated using the pseudo-label set and sharpened classification predictions. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further refined. This classification model is further trained; an integrated pseudo-label set is obtained based on the further trained multi-label text classification model; the adjacency matrix of the k-nearest neighbor graph is determined based on the latent text features learned by the m-th layer encoder; the label correlation matrix is ​​determined based on the parameter matrix of the encoder layer; noise correction is performed on the integrated pseudo-label set based on the adjacency matrix and the label correlation matrix; the further trained multi-label text classification model is then finally trained based on the noise-corrected pseudo-label set to obtain the trained multi-label text classification model; the text to be classified is input into the trained multi-label text classification model to obtain the label corresponding to the text to be classified. This invention proposes a novel weakly supervised multi-label text classification method that can learn a reliable multi-label text classification model using only a small number of category words as supervision information and a small amount of unlabeled text. It includes a novel correlation-based supervision information propagation method that can use text nearest neighbor relationships and label correlations to propagate noisy supervision information to correct the noisy supervision information, thereby improving classification accuracy. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a label propagation text classification method that integrates weakly supervised information, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall framework for weakly supervised text classification learning provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a category word generation method (SRC) provided in an embodiment of the present invention. Figure 4 This is a flowchart illustrating a weakly supervised multi-label text classification method (CLP) provided in an embodiment of the present invention. Figure 5This is a block diagram of a tag propagation text classification device that integrates weakly supervised information and is provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a tag propagation text classification device that integrates weakly supervised information generated according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0013] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0014] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0015] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0016] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0017] This invention provides a method for classifying label propagation text by fusing weakly supervised information. This method can be implemented by a label propagation text classification device that fuses weakly supervised information, which can be a terminal or a server. Figure 1 The flowchart shown is for a label propagation text classification method that integrates weakly supervised information. The processing flow of this method may include the following steps:

[0018] S1. Select an initial set of category words, and pre-train the initial multi-label text classification model based on the initial set of category words to obtain a pre-trained multi-label text classification model. The multi-label text classification model includes an encoder layer and a prediction layer.

[0019] In one feasible implementation, a fundamental task in many practical applications is to build a multi-label text classification model that can predict multiple relevant labels for unseen raw text. However, in this case, there is only a set of unlabeled raw text samples. and a set of predefined label names Available, where the original text follows everyday language conventions, including appropriate capitalization, punctuation, and readable formatting. One feasible approach is to first select a set of category words for each category. As weakly supervised information, then only used and We apply a weakly supervised learning method for multi-label text classification to construct a multi-label text classification model.

[0020] Following this line of thought, this invention proposes a novel weakly supervised learning framework for multi-label text classification. For example... Figure 2 As shown, the framework consists of two parts: a Significance Ranking and Crowd-sourcing (SRC) method and a weakly supervised learning multi-label text classification method (Correlation-aware Label Propagation, CLP). Specifically, SRC generates reliable category words by manually selecting weighted words from candidate category words generated by topic modeling. As a weak supervisor; then, given and CLP corrects the weak supervision provided by category words through relevance-aware supervised propagation, thereby learning a reliable multi-label text classification model. SRC and CLP will be introduced in more detail below. It should be noted that in this invention, the concepts of label and category are the same concept used in different contexts.

[0021] The proposed SRC method is introduced below, which is used to generate a set of category words. Similar to previous studies, category words can be manually selected from the list of topic words generated by the topic model. However, some words may appear in the top word lists of multiple topics, which makes manual selection difficult. Therefore, in SRC, the topic words are first reordered to filter out duplicate and unimportant words before manual selection. The detailed process is as follows... Figure 3 As shown.

[0022] In SRC, topic models are used to learn latent topics from unlabeled text, with each topic derived from a multinomial distribution over the vocabulary. This indicates that, to maintain a one-to-one correspondence between topics and categories, SeededLDA is used with seed words. From the text Inferring the topic from the middle. For the first Each category selects category words from the most important words corresponding to the learned topics. Optionally, S1 selects an initial set of category words, including:

[0023] S11. Select the top 100 keywords for each category as early candidate category keywords.

[0024] In one feasible implementation, the top 100 keywords of the category are selected. As early candidate category words. In standard topic modeling, topic weights... This reflects the importance of a word to its corresponding topic. Generally, a higher weight indicates that the word is more descriptive of the topic. Therefore, top-ranking topic words are selected as candidate category words based on their topic weights.

[0025] S12. Calculate the TW-ITF value of each topic word by multiplying the topic weight of each topic word by the inverse topic frequency, and reorder the topic words in the early candidate category words according to the TW-ITF value of each topic word.

[0026] In one feasible implementation, due to the co-occurrence of words in a text, topic models may generate duplicate topic terms for different topics, a phenomenon known as topic term redundancy. In multi-label text classification scenarios, this phenomenon is exacerbated by category correlation, leading to significant initial pseudo-label noise and severely impacting classification performance. To filter out unimportant words, we propose a new scoring metric—Topic Weight-Inverse Topic Frequency (TW-ITF)—to reorder candidate category words, denoted as... Inspired by TF-IDF, it is argued that words appearing in more topics are often less important. Therefore, TW-ITF is calculated by multiplying the topic weight of each word by its inverse topic frequency:

[0027]

[0028] in, Representing words The weights; Representing words The topic frequency (i.e., how many topics the word appears in as a candidate category word) can filter out words that tend to appear repeatedly across different topics. In this way, words with high topic weight and low topic frequency rank higher, making category words exclusive to their corresponding categories.

[0029] After sorting, a crowdsourcing platform (https: / / www.upwork.com / ) can be used to manually select the final category words from the re-sorted candidate category words. One feasible implementation is to invite 10 volunteers to select the final category words based on their personal judgment. Volunteers can choose up to 10 words. They are encouraged to choose words that meet the following criteria: i) high TW-ITF score; ii) able to express the meaning of the category; and iii) exclusive to each category.

[0030] The following quality control strategies were implemented: i) Recruit volunteers with relevant professional backgrounds and provide examples of reference category terms; ii) Measure the redundancy of selected terms using unnormalized topic uniqueness, encouraging volunteers with high redundancy to choose more exclusive terms; iii) Have experts review highly controversial terms; iv) When aggregating results, only retain category terms selected by at least 6 volunteers. Through the above process, the selected keywords are both representative and exclusive to each category, thus obtaining pseudo-labels with less noise and improving classification performance.

[0031] Repeat the above process for all categories to obtain the final set of category words. .

[0032] like Figure 3 As shown in the example, the category names of the original text could be "1.coffee 2.sugar 3.money-fx 4.trade", and the initially generated list of keywords is shown in Table 1 below: Table 1

[0033] After reordering using TW-ITF, the category terms in Table 2 below can be obtained through manual selection: Table 2

[0034] The above steps propose a novel method for selecting category words, which can generate representative category words and produce relatively reliable weakly supervised information for learning classification models. This includes a novel category word importance ranking method that can effectively filter the importance of words belonging to multiple categories.

[0035] S2. Input the original text into the encoder layer to obtain deep latent features, and input the deep latent features into the prediction layer to obtain the initial classification prediction results.

[0036] One feasible implementation utilizes unlabeled training raw text samples. and the set of category words generated by SRC We propose a weakly supervised learning method, CLP, for multi-label text classification, to build multi-label text classification models.

[0037] Optionally, the encoder layer is a layer that includes The pre-trained BERT layer uses express, This represents the encoder parameters. The prediction layer is a multilayer perceptron with a sigmoid activation function. express, This represents the parameters of the prediction layer.

[0038] In one feasible implementation, the structure of a multi-label text classification model based on BERT fine-tuning is first introduced. Specifically, a model containing... Pre-trained BERT layers as encoder And add a multilayer perceptron (MLP) with a sigmoid activation function as the prediction layer. ,in and These represent the encoder parameters and the prediction layer parameters, respectively. In this invention, the BERT encoder serves as the backbone network, but it can also be replaced by other deep neural network structures.

[0039] S2 inputs the original text into the encoder layer to obtain deep latent features, and then inputs these deep latent features into the prediction layer to obtain initial classification prediction results, including: S21. For each original text According to the following formula (1), from all of the encoder Deep latent features obtained from layers : (1) S22. Input the deep latent features into the prediction layer, and obtain the initial classification prediction result according to the following formula (2): (2).

[0040] In one feasible implementation, for each original text From all of the encoders Deep latent features obtained from layers And the final classification prediction results The calculation process is shown in equations (1)-(2) above. Multi-label text classification model It can utilize a set of category words The generated pseudo-labels are used for training.

[0041] S3. Determine the set of pseudo-tags based on the frequency of category words in the original text.

[0042] In one feasible implementation, the above steps utilize a set of category words. The generated pseudo-labels are used for training; however, these pseudo-labels contain a large amount of noise, making it difficult to train a reliable multi-label text classification model. Therefore, a warm-up objective function is first designed to update the noisy pseudo-labels. Subsequently, during model training, these noisy supervision signals are further corrected by propagating the pseudo-labels on the text correlation graph. The overall process of the proposed weakly supervised learning multi-label text classification method CLP is as follows: Figure 4 As shown.

[0043] Given a set of category words We can use statistical category words in text The pseudo-tags for each text are initialized based on the number of occurrences in the text. :

[0044] in, Indicates category All category words in the text The total number of times it appears in; This represents the threshold used to filter out noisy signals. All text with pseudo-label metrics set to zero is removed, resulting in a set of pseudo-labeled samples. ,in This indicates that there is at least one set of text indices that initialize pseudo-labels.

[0045] S4. The pseudo-label set is updated step by step using the pseudo-label set and sharpened classification prediction. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained.

[0046] In one feasible implementation, a pseudo-label sample set can be used. Training the model As a warm-up step, these pseudo-labels, i.e., weakly supervised annotations, often contain noise, which can severely affect classification performance. To alleviate this problem, this invention draws on the idea of ​​weakly supervised self-training methods, combining the current pseudo-labels during model training. Compared with sharpened classification prediction We will gradually update the pseudo-tags.

[0047] Optionally, S4 progressively updates the pseudo-label set using the pseudo-label set and sharpened classification predictions. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained, including: S41. According to the following formula (3), the smoothing parameter is adaptively adjusted by applying exponentially increasing annealing. : (3) in, and These are pre-set Initial value and final value, It represents the total number of training epochs.

[0048] S42, Based on smoothing parameters The initial classification prediction results and the following formula (4) are used to calculate the sharpened classification prediction. Each component: (4) in, This represents the predicted probability value of the i-th sample in the j-th category. This represents the predicted probability value of the i'th sample in the j-th category.

[0049] S43, By combining the current pseudo-tag Compared with sharpened classification prediction To gradually update the pseudo-tags, as shown in equation (5): (5) S44. Based on the updated pseudo-label set, the initial classification prediction results, the binary cross-entropy loss, and the following formula (6), the pre-trained multi-label text classification model is further trained: (6) in, This represents the binary cross-entropy loss.

[0050] S5. Obtain the integrated pseudo-label set based on the further trained multi-label text classification model.

[0051] In one feasible implementation, pseudo-labels are propagated on the text relevance graph during this stage, and the model is further optimized in conjunction with model training. .

[0052] Each sample is re-labeled with a pseudo-label based on the model's predictions. Specifically, all training text samples... The pseudo-labels are marked as follows:

[0053]

[0054] in, (like () represents the pseudo-labels obtained after the model training is completed; This represents the sharpened classification prediction result generated by the preheated multi-label text classification model.

[0055] S6. Determine the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder.

[0056] In one feasible implementation, the re-labeled pseudo-tags Noise still exists and can severely impact classification performance. To address this issue, given a set of pseudo-labeled samples... In this case, the multi-label text classification model is further improved by performing supervised propagation on the relevance-aware text graph. Specifically, neighboring samples of text often have similar labels, and related labels tend to co-occur in the same text. Therefore, a text relevance graph is constructed by analyzing text neighbors and label relevance, and noisy pseudo-labels are corrected by propagating pseudo-labels on this graph.

[0057] Optionally, S6 determines the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder, including: Latent text features learned from the m-th layer encoder And the following equation (7) determines the adjacency matrix. : (7) in, express The neighbor set of the matrix, where T represents the transpose of the matrix. This represents the deep latent features obtained from one layer of the encoder, as described above. It is obtained from each layer in layer k. The sum, This indicates that the i-th sample uses the feature output obtained from the m-th layer of the encoder. This represents the feature output obtained by using the m-th layer of the encoder for the j-th sample.

[0058] In one feasible implementation, a -Nearest Neighbor Graph ,in and Let represent the set of text nodes and the set of edges between text neighbors, respectively. Their adjacency matrix is... By the Latent text features learned by layer encoder The result is obtained as shown in equation (7) above.

[0059] along with The update will also be updated. Furthermore, to improve efficiency, readily available approximations are used. -NN search solver, this tool can perform high-speed nearest neighbor search approximation on a large amount of data in a high-dimensional vector space, making the method feasible when processing massive text samples.

[0060] S7. Determine the tag correlation matrix based on the parameter matrix of the encoder layer.

[0061] In one feasible implementation, in order to refine pseudo-labels while taking into account label relevance, we introduce a label relevance matrix through prediction parameters.

[0062] Optionally, S7 determines the tag correlation matrix based on the parameter matrix of the encoder layer, including: The columns of the parameter matrix of the encoder layer As a representation of each category, the label relevance matrix S is calculated using cosine similarity and the following equation (8): (8) in, Indicates to each column Normalized vector; represents element-wise division, and T represents matrix transpose.

[0063] In one feasible implementation, the columns of the prediction parameter matrix are... As a representation of each category, the label relevance matrix is ​​estimated using the cosine similarity between them. : To accommodate the matrix-based update process in subsequent model training, the above calculations are converted into the form of the above formula (8).

[0064] S8. Based on the adjacency matrix and the label correlation matrix, noise correction is performed on the integrated pseudo-label set. The multi-label text classification model is then trained using the noise-corrected pseudo-label set to obtain the trained multi-label text classification model.

[0065] In one feasible implementation, samples typically tend to share similar labels with their nearest neighbors. In multi-label scenarios, related labels also tend to be assigned to the same text, such as "market" and "stock market". Therefore, this invention proposes to correct noisy supervision information during model training by propagating pseudo-labels on the text nearest neighbor adjacency matrix A and the label correlation matrix S.

[0066] Alternatively, the specific operation of S8 can be as follows: Through the adjacency matrix Tag correlation matrix And the following equation (9) is used to correct noise in the integrated pseudo-label set: (9) in, A pseudo-label matrix representing all text; express The symmetric normalized matrix; express The angle matrix; The propagation rate is used to avoid introducing noise supervision from text neighbors and estimated label relevance. Pseudo-label propagation can enhance the supplementation of missing labels and correct noisy labels based on text neighbors and label relevance.

[0067] S9. Input the text to be classified into the trained multi-label text classification model to obtain the label corresponding to the text to be classified.

[0068] In one feasible implementation, in order to simultaneously avoid noise and emphasize accurate label relevance, it is assumed that the estimated label relevance matrix... The label correlation matrix S should be sparse, and the prediction parameter matrix W should be low-rank. Therefore, sparsity and low-rank constraints are introduced on the prediction parameters to enhance parameter learning. Specifically, the sparsity constraint on the label correlation matrix S helps filter out noisy label correlations, while the low-rank constraint on the prediction parameters W introduces more correlations between class representations. These two constraints complement each other, resulting in more accurate label correlations by simultaneously enhancing true correlations and filtering out inaccurate correlations.

[0069] Alternatively, the specific operation of S9 can be as follows: The objective function is constructed based on the noise-corrected pseudo-label set and the following equation (10): (10) in, express Norm; Represents the nuclear norm; This represents the scaling parameter.

[0070] In this embodiment of the invention, an initial set of category words is selected, and an initial multi-label text classification model is pre-trained based on the initial set of category words to obtain a pre-trained multi-label text classification model. The multi-label text classification model includes an encoder layer and a prediction layer. The original text is input into the encoder layer to obtain deep latent features, and the deep latent features are input into the prediction layer to obtain the initial classification prediction result. A pseudo-label set is determined based on the frequency of category words in the original text. The pseudo-label set is gradually updated using the pseudo-label set and sharpened classification prediction. Based on the updated pseudo-label set, the initial classification prediction result, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further refined. The multi-label text classification model is further trained; an integrated pseudo-label set is obtained based on the further trained multi-label text classification model; the adjacency matrix of the k-nearest neighbor graph is determined based on the latent text features learned by the m-th layer encoder; the label correlation matrix is ​​determined based on the parameter matrix of the encoder layer; noise correction is performed on the integrated pseudo-label set based on the adjacency matrix and the label correlation matrix; the further trained multi-label text classification model is then finally trained based on the noise-corrected pseudo-label set to obtain the trained multi-label text classification model; the text to be classified is input into the trained multi-label text classification model to obtain the label corresponding to the text to be classified. This invention proposes a novel weakly supervised multi-label text classification method that can learn a reliable multi-label text classification model using only a small number of category words as supervision information and a small amount of unlabeled text. It includes a novel correlation-based supervision information propagation method that can propagate noisy supervision information using text nearest neighbor relationships and label correlations to correct the noisy supervision information, thereby improving classification accuracy.

[0071] Figure 5 This is a block diagram of a label propagation text classification device that integrates weakly supervised information, provided in an embodiment of the present invention. This device is used in a label propagation text classification method that integrates weakly supervised information. (Refer to...) Figure 3 The device includes:

[0072] The pre-training module 510 is used to select an initial set of category words and pre-train the initial multi-label text classification model based on the initial set of category words to obtain a pre-trained multi-label text classification model, wherein the multi-label text classification model includes an encoder layer and a prediction layer. The feature extraction module 520 is used to input the original text into the encoder layer to obtain deep latent features, and input the deep latent features into the prediction layer to obtain the initial classification prediction result; The statistics module 530 is used to determine the set of pseudo-tags based on the frequency of category words in the original text; The update module 540 is used to progressively update the pseudo-label set using the pseudo-label set and sharpened classification predictions, and further train the pre-trained multi-label text classification model based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss. Integration module 550 is used to obtain an integrated set of pseudo-labels based on a further trained multi-label text classification model; The first determining module 560 is used to determine the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder. The second determining module 570 is used to determine the tag correlation matrix based on the parameter matrix of the encoder layer; Training module 580 is used to correct noise in the integrated pseudo-label set based on the adjacency matrix and label correlation matrix, and to finally train the multi-label text classification model based on the noise-corrected pseudo-label set to obtain the trained multi-label text classification model. The classification module 590 is used to input the text to be classified into a trained multi-label text classification model to obtain the label corresponding to the text to be classified.

[0073] Figure 6 This is a schematic diagram of the structure of a label propagation text classification device that integrates weakly supervised information generated according to an embodiment of the present invention, as shown below. Figure 6 As shown, a label propagation text classification device that integrates weakly supervised information can include the above-mentioned... Figure 5 The illustrated label propagation text classification device 610, which integrates weakly supervised information to generate labels, may optionally include a first processor 2001.

[0074] Optionally, the label propagation text classification device 610, which integrates weakly supervised information to generate labels, may also include a memory 2002 and a transceiver 2003.

[0075] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0076] The following is combined with Figure 6 The following is a detailed description of the various components of the label propagation text classification device 610 generated by fusing weakly supervised information: The first processor 2001 is the control center of the tag propagation text classification device 610 generated by fusing weakly supervised information. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0077] Optionally, the first processor 2001 can execute various functions of the label propagation text classification device 610 generated by fusing weakly supervised information by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0078] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 are shown in the diagram.

[0079] In a specific implementation, as one example, the label propagation text classification device 610 generated by fusing weakly supervised information may also include multiple processors, such as... Figure 6 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0080] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0081] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently, and may be connected to the interface circuit of the text classification device 610 via tags generated by fusing weakly supervised information. Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0082] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0083] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 6 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0084] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be transmitted through the interface circuit of the text classification device 610 via tags generated by fusing weakly supervised information. Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0085] It should be noted that, Figure 6 The structure of the label propagation text classification device 610 shown in the figure does not constitute a limitation on the router. Actual label propagation text classification devices that integrate weakly supervised information may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0086] Furthermore, the technical effect of the label propagation text classification device 610 that integrates weakly supervised information can be referred to the technical effect of the label propagation text classification method that integrates weakly supervised information as described in the above method embodiments, and will not be repeated here.

[0087] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor.

[0088] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0089] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0090] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0091] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0092] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0098] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A text classification method that integrates weakly supervised information to generate labels propagating from text, characterized in that, The method includes: S1. Select an initial set of category words, and pre-train the initial multi-label text classification model based on the initial set of category words to obtain a pre-trained multi-label text classification model, which includes an encoder layer and a prediction layer. S2. Input the original text into the encoder layer to obtain deep latent features, and input the deep latent features into the prediction layer to obtain the initial classification prediction result; S3. Determine the set of pseudo-tags based on the frequency of category words in the original text; S4. The pseudo-label set is updated step by step using the pseudo-label set and sharpened classification prediction. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained. S5. Obtain the integrated pseudo-label set based on the further trained multi-label text classification model; S6. Determine the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder. S7. Determine the tag correlation matrix based on the parameter matrix of the encoder layer; S8. Based on the adjacency matrix and the label correlation matrix, noise correction is performed on the integrated pseudo-label set. Based on the noise-corrected pseudo-label set, the multi-label text classification model is trained for the final training to obtain the trained multi-label text classification model. S9. Input the text to be classified into the trained multi-label text classification model to obtain the label corresponding to the text to be classified.

2. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, The initial set of category words selected in S1 includes: S11. Select the top 100 keywords for each category as early candidate category keywords; S12. Calculate the TW-ITF value of each topic word by multiplying the topic weight of each topic word by the inverse topic frequency, and reorder the topic words in the early candidate category words according to the TW-ITF value of each topic word.

3. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, The encoder layer is included The pre-trained BERT layer uses express, Indicates encoder parameters; the prediction layer is a multilayer perceptron with a sigmoid activation function, employing... express, Indicates the parameters of the prediction layer; The process S2 involves inputting the original text into the encoder layer to obtain deep latent features, and then inputting these deep latent features into the prediction layer to obtain initial classification prediction results, including: S21. For each original text According to the following formula (1), from all of the encoder Deep latent features obtained from layers : （1） S22. Input the deep latent features into the prediction layer, and obtain the initial classification prediction result according to the following formula (2): （2）。 4. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, S4 involves progressively updating the pseudo-label set using the pseudo-label set and sharpened classification predictions. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained, including: S41. According to the following formula (3), the smoothing parameter is adaptively adjusted by applying exponentially increasing annealing. : （3） in, and These are pre-set Initial value and final value, It is the total number of training epochs; S42, Based on smoothing parameters The initial classification prediction results and the following formula (4) are used to calculate the sharpened classification prediction. Each component: （4） in, This represents the predicted probability value of the i-th sample in the j-th category. This represents the predicted probability value of the i'th sample in the j-th category; S43, By combining the current pseudo-tag Compared with sharpened classification prediction To gradually update the pseudo-tags, as shown in equation (5): （5） S44. Based on the updated pseudo-label set, the initial classification prediction results, the binary cross-entropy loss, and the following formula (6), the pre-trained multi-label text classification model is further trained: （6） in, This represents the binary cross-entropy loss.

5. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, The step S6, which determines the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned from the m-th layer encoder, includes: Latent text features learned from the m-th layer encoder And the following equation (7) determines the adjacency matrix. : （7） in, express The neighbor set of the matrix, where T represents the transpose of the matrix. This indicates that the i-th sample uses the feature output obtained from the m-th layer of the encoder. This represents the feature output obtained by using the m-th layer of the encoder for the j-th sample.

6. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, The step S7, determining the tag correlation matrix based on the parameter matrix of the encoder layer, includes: The columns of the parameter matrix of the encoder layer As a representation of each category, the label relevance matrix S is calculated using cosine similarity and the following equation (8): （8） in, Indicates to each column Normalized vector; represents element-wise division, and T represents matrix transpose.

7. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, S8 performs noise correction on the integrated pseudo-label set based on the adjacency matrix and the label correlation matrix, including: Through the adjacency matrix Tag correlation matrix And the following equation (9) is used to correct noise in the integrated pseudo-label set: （9） in, A pseudo-label matrix representing all text; express The symmetric normalized matrix; express The angle matrix; Indicates the transmission rate.

8. The label propagation text classification method based on weakly supervised information as described in claim 1, characterized in that, S9 performs final training on the multi-label text classification model based on the noise-corrected pseudo-label set, resulting in a trained multi-label text classification model, including: The objective function is constructed based on the noise-corrected pseudo-label set and the following equation (10): （10） in, express Norm; Represents the nuclear norm; This represents the scaling parameter.

9. A label propagation text classification device that integrates weakly supervised information, wherein the label propagation text classification device that integrates weakly supervised information is used to implement the label propagation text classification method that integrates weakly supervised information as described in any one of claims 1-8, characterized in that, The device includes: The pre-training module is used to select an initial set of category words and pre-train the initial multi-label text classification model based on the initial set of category words to obtain a pre-trained multi-label text classification model, which includes an encoder layer and a prediction layer. The feature extraction module is used to input the original text into the encoder layer to obtain deep latent features, and then input the deep latent features into the prediction layer to obtain the initial classification prediction results. The statistics module is used to determine the set of pseudo-tags based on the frequency of category words in the original text; The update module is used to progressively update the pseudo-label set using the pseudo-label set and sharpened classification predictions. Based on the updated pseudo-label set, the initial classification prediction results, and the binary cross-entropy loss, the pre-trained multi-label text classification model is further trained. The integration module is used to obtain an integrated set of pseudo-labels based on the further trained multi-label text classification model. The first determining module is used to determine the adjacency matrix of the k-nearest neighbor graph based on the latent text features learned by the m-th layer encoder. The second determining module is used to determine the tag correlation matrix based on the parameter matrix of the encoder layer; The training module is used to correct the noise of the integrated pseudo-label set based on the adjacency matrix and the label correlation matrix, and to finally train the multi-label text classification model based on the noise-corrected pseudo-label set to obtain the trained multi-label text classification model. The classification module is used to input the text to be classified into a trained multi-label text classification model to obtain the label corresponding to the text to be classified.

10. A label propagation text classification device that integrates weakly supervised information generation, characterized in that, The label propagation text classification device generated by fusing weakly supervised information includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 8.

Images