A hierarchical text classification method and system based on differentiated prompt characters

Abstract

The application provides a hierarchical text classification method and system based on differential prompt characters, relates to the technical field of text classification, and designs a hierarchical prediction prompt template with multiple placeholders; semantic feature extraction and aggregation are respectively performed on a first-level category set and a second-level category set to obtain differential prompt characters of the first level and the second level; the text to be classified, the differential prompt characters of the first level and the second level are filled into corresponding placeholders of the hierarchical prediction prompt template to obtain a to-be-predicted sequence; and the to-be-predicted sequence is input into a pre-trained language model to predict first-level category masks and second-level category masks, so that a final prediction result is obtained. Based on the designed hierarchical prediction prompt template, the application uses prompt learning, converts the text classification problem into a completion gap filling problem in a small sample background, and realizes the hierarchical text classification task based on knowledge mining of the pre-trained language model.

Description

Technical Field

[0001] This invention belongs to the field of text classification technology, and in particular relates to a hierarchical text classification method and system based on differentiated prompt characters. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] For a comprehensive database of professional articles covering multiple fields, it is difficult to accurately distinguish which major category and subcategory each article belongs to due to the involvement of multiple professional fields. Therefore, using deep learning to achieve automated hierarchical text classification is extremely promising and valuable in terms of development and application. Hierarchical text classification, as shown in the example, is... Figure 1 As shown, an article belongs to a major category in the first level, and is further subdivided into a minor category in the second level.

[0004] In today's reality, manually classifying massive amounts of articles is impractical. Therefore, the problem to be solved is how to achieve relatively accurate hierarchical text classification in small sample scenarios. Currently, a method for hierarchical text classification of articles has been implemented through fine-tuning. However, according to experimental results, it cannot achieve hierarchical text classification well for small samples. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a hierarchical text classification method and system based on differentiated prompt characters. Based on the designed hierarchical prediction prompt template, prompt learning is used to transform the text classification problem into a cloze test problem in the context of small samples. Based on knowledge mining of the pre-trained language model, the hierarchical text classification task is achieved.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention provides a hierarchical text classification method based on differentiated prompt characters.

[0008] A hierarchical text classification method based on differentiated prompt characters includes:

[0009] Design a hierarchical prediction prompt template with multiple pre-defined placeholders, wherein the placeholders include: text to be classified, first-level prompt character, first-level category mask, second-level prompt character, and second-level category mask;

[0010] Semantic features are extracted and aggregated from the first-level category set and the second-level category set respectively to obtain differentiated prompt characters for the first and second levels;

[0011] Fill the corresponding placeholders in the hierarchical prediction prompt template with the text to be classified, the differentiated prompt characters of the first level and the second level, to obtain the sequence to be predicted;

[0012] The sequence to be predicted is input into a pre-trained language model to predict the first-level class mask and the second-level class mask, thus obtaining the final prediction result.

[0013] Furthermore, the hierarchical prediction prompt template is as follows:<original_input> <t0> <mask0> <t1> <mask1>,in, <mask0> 、 <mask1>These are the first-level category mask and the second-level category mask, respectively. <t0> 、 <t1>These represent the first-level prompt characters and the second-level prompt characters, respectively, while original_input represents the text to be categorized.

[0014] Furthermore, TextCNN is used to extract and aggregate semantic features from the first-level and second-level category sets respectively, resulting in differentiated prompt characters for the first and second levels, which are then used as hierarchical prediction prompt templates. <t0> 、 <t1>The fill value.

[0015] Furthermore, the semantic feature extraction and aggregation specifically involves the following steps:

[0016] Each category in the category set is input into the pre-trained language model's embedding mechanism and a custom linear layer to generate an intermediate vector for each category;

[0017] Hierarchical feature extraction and aggregation are performed on the intermediate vector to obtain the aggregated feature vector.

[0018] Furthermore, the custom linear layer is a linear transformation that normalizes the regression values ​​of the categories.

[0019] Furthermore, the pre-trained language model is used to introduce the prediction sequence to the model, and the result tensor is obtained through the MLM Head.

[0020] Furthermore, the final prediction result is obtained by selecting the regression values ​​of the first-level category mask and the second-level category mask from the result tensor, obtaining the probability of each level category in the regression value, then synthesizing multiple probabilities through the linear layer of the corresponding category, and finally finding the index of the maximum value to obtain the first-level category and the second-level category.

[0021] A second aspect of the present invention provides a hierarchical text classification system based on differentiated prompt characters.

[0022] A hierarchical text classification system based on differentiated prompt characters includes a template design module, a category differentiation module, a template filling module, and a category prediction module.

[0023] The template design module is configured to: design a hierarchical prediction prompt template with multiple pre-defined placeholders, wherein the placeholders include: text to be classified, first-level prompt character, first-level category mask, second-level prompt character, and second-level category mask;

[0024] The category differentiation module is configured to extract and aggregate semantic features from the first-level category set and the second-level category set respectively, to obtain differentiated prompt characters for the first and second levels.

[0025] The template filling module is configured to fill the text to be classified and the differential prompt characters of the first and second levels into the corresponding placeholders of the hierarchical prediction prompt template to obtain the sequence to be predicted;

[0026] The category prediction module is configured to input the sequence to be predicted into the pre-trained language model PLM, predict the first-level category mask and the second-level category mask, and obtain the final prediction result.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of a hierarchical text classification method based on differentiated prompt characters as described in the first aspect of the present invention.

[0028] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a hierarchical text classification method based on differentiated prompt characters as described in the first aspect of the present invention.

[0029] The above one or more technical solutions have the following beneficial effects:

[0030] This invention utilizes cue learning and designs a template to predict the first and second level categories respectively. In the context of small sample sizes, the text classification problem is transformed into a cloze test problem. Using a pre-trained language model, a hierarchical text classification task is achieved. Features are extracted from each category and aggregated into a vector as a cue token for predicting that level. A linear layer is designed for each category to cleverly solve the problem of how to aggregate data before convolution operations and during the mapping from vocabulary to category if a category can form multiple token IDs, thereby improving the prediction accuracy and robustness of the model.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 A diagram illustrating hierarchical text classification.

[0034] Figure 2 This is a flowchart of the method in the first embodiment.

[0035] Figure 3 This is a schematic diagram of the classification model for the first embodiment.

[0036] Figure 4 This is the first embodiment. <t0>Generate a graph.

[0037] Figure 5 This is the first embodiment. <t1>Generate a graph. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] In the context of small sample sizes, this invention utilizes cue learning to transform the text classification problem into a cloze test problem. Because the data format of cloze tests better matches the task of the pre-trained language model during the pre-training phase, it can better extract the knowledge acquired by the pre-trained language model. To further extract the knowledge acquired by the pre-trained language model and achieve more accurate text classification, this invention proposes a novel template construction method using hierarchical prediction cue templates.<original_input> <t0> <mask> <t1> <mask>,in, <t0> 、 <t1>These represent the first and second layer hint tokens, respectively; they achieve good results in the Web-Of-Science (WOS) dataset.

[0041] Example 1

[0042] In one or more embodiments, a hierarchical text classification method based on differentiated prompt characters is disclosed, such as... Figure 2 As shown, it includes the following steps:

[0043] Step S1: Design a hierarchical prediction prompt template with multiple pre-set placeholders, wherein the placeholders include: text to be classified, first-level prompt character, first-level category mask, second-level prompt character, and second-level category mask;

[0044] Step S2: Extract and aggregate semantic features from the first-level category set and the second-level category set respectively to obtain differentiated prompt characters for the first and second levels;

[0045] Step S3: Fill the corresponding placeholders in the hierarchical prediction prompt template with the text to be classified, the differentiated prompt characters of the first level and the second level, to obtain the sequence to be predicted;

[0046] Step S4: Input the sequence to be predicted into the pre-trained language model, predict the first-level category mask and the second-level category mask, and obtain the final prediction result.

[0047] The following is a detailed description of the implementation process of a hierarchical text classification method based on differentiated prompt characters in this embodiment.

[0048] This embodiment aims to solve two practical problems:

[0049] (1). How to design a prompt template for hierarchical text classification tasks in the case of small sample size, so that it can better mine and utilize the knowledge acquired by the pre-trained language model in the pre-training stage to achieve more accurate prediction.

[0050] (2). After the sequence data filled with template is input into the pre-trained language model, it passes through the MLM Head. For each data point at each level, a vector of [1, vocab_size] is generated to form the result tensor. In order to realize the mapping from vocabulary to category, assuming that the words of each category are used as the mapping vocabulary, if the number of token IDs generated for the corresponding word is greater than 1, how to realize the mapping from the probability values ​​corresponding to multiple token IDs to the final class unique probability value.

[0051] To address the two issues mentioned above, this embodiment provides a hierarchical text classification method that constructs hierarchical prediction prompt templates using differentiated prompt characters, such as... Figure 3 As shown, the filled hierarchical prediction prompt template is input into the trained classification model BertForMaskedLM for pre-trained language model knowledge mining. Under limited sample size, a hierarchical text classification task is achieved, mainly including the following steps:

[0052] Step 1: Design a hierarchical prediction prompt template

[0053] The template is in the form of<original_input> <t0> <mask0> <t1> <mask1>This includes five placeholders: original_input represents the text to be categorized, <mask0> 、 <mask1>These are the first-level category mask and the second-level category mask, respectively. <t0> 、 <t1>These represent the first-level prompt character and the second-level prompt character, respectively.

[0054] Step 2: Use TextCNN to extract and aggregate semantic features from the category sets at each level, and obtain... <mask0> 、 <mask1>The preceding differentiated prompt characters <t0> 、 <t1>,like Figure 4 , 5 As shown, specifically:

[0055] Step 2.1: Design a linear layer for each category to perform a linear transformation that normalizes the regression values ​​of the category. Each initial weight of the linear transformation is 1 / the number of token IDs in the category, with a bias of 0. For example, if the number of token IDs in a category is 2, then the final class embedding is...

[0056] Step 2.2: Introduce each category word into the pre-trained language model's embedding mechanism and a custom linear layer, ensuring that each category can form an intermediate vector of size [1, hidden_size], where hidden_size represents the length of the hidden layer vector, i.e.:

[0057] label emb =embedding(label) token (1)

[0058] label emb =linear[label](label) emb (2)

[0059] Where, label token It is a collection of IDs formed by the encoding function of the tokenizer for a certain category. label represents a category, and linear[label] is the linear layer of the label for the specific category.

[0060] Step 2.3: Based on the operations in Step 2.2, perform a linear transformation on the categories of the first and second levels to obtain label0_emb and label1_emb, where... label0_num and label1_num are the number of categories in the first and second levels, respectively.

[0061] The labels 0_emb and 1_emb are respectively introduced into the corresponding convolutional layers to achieve hierarchical feature extraction and aggregation, resulting in an aggregated feature vector, which can be expressed by the formula:

[0062]

[0063]

[0064] Among them, CNN level0 The width of the convolutional kernel is [1, 3, 5], CNN level1 The width of the convolution kernel is [1, 3, 5, 7].

[0065] Finally, the aggregated feature vector obtained after TextCNN convolution is processed, and then concatenated using the cat operation. They were used as differentiating prompts respectively. <t0> 、 <t1>The padding values ​​respectively guide the mask following the prompt. <mask0> 、 <mask1>Make accurate predictions.

[0066] Step 3: Convert the original input sequence x = (x0, x1, ... x... n Differentiation prompt <t0> 、 <t1>Fill into template<original_input> <t0> <mask> <t1> <mask>The new input sequence x is obtained from the corresponding placeholders. p = <x> <t0> <mask> <t1> <mask>Wherein, the original input sequence x is the sequence of text to be classified after being encoded by the symbolizer.

[0067] Step 4: Convert the new input sequence x p Introducing a pre-trained language model M, and obtaining the resulting tensor logits through an MLM Head:

[0068] logits = M(x p (5)

[0069] in, Select <masko>and <mask1>The regression values ​​are named logit_0 and logit_1, respectively.

[0070] Step 5: Select the probabilities of the first-level category and the second-level category in logit_0 and logit_1 respectively, and introduce the corresponding linear layers of the category to synthesize multiple probabilities.

[0071] Specifically, for each category y, first find the regression value of logit_0 / logit_1 at the token id position, then substitute it into the linear layer corresponding to the category to obtain the probability of being predicted as that category. For example, if the regression values ​​at the token id position for category y are 0.15 and 0.52 respectively, the weights of the linear layer are [0.48, 0.53], and the bias is 0.001, then the probability of that category being predicted is... Therefore, two vectors, logit0 and logit1, will be generated in the end. By using linear layers—a parameter-optimizable method—to integrate a certain type of probability, rather than using averaging, the predictive accuracy and robustness of the model are improved.

[0072] Step 6: Obtain the first <mask>Position and the second <mask>After obtaining the regression values ​​for the locations, the index of the maximum value is calculated to obtain the final prediction result.

[0073] pred0 = argmax(logit0) (6)

[0074] pred1 = argmax(logit1) (7)

[0075] Wherein, pred0 and pred1 are the predicted first-level category and the first-level category, respectively.

[0076] During the model training phase, the loss is calculated for the regression values ​​at each level using Cross_Entropy_Loss, specifically as follows:

[0077] loss0=Cross_Entropy-Loss(logit0, label0) (8)

[0078] loss1=Cross_Entropy_Loss(logit1, label1) (9)

[0079] loss = loss0 + loss1 (10)

[0080] Where Cross_Entropy_Loss is the cross-entropy loss function, logit0 is the probability vector of the first-level class, and label0 is the vector composed of the true values ​​of the first-level class. logit1 is the probability vector of the second-level class, and label1 is the vector composed of the true values ​​of the second-level class. loss0 is the loss of the first level, loss1 is the loss of the second level, and loss is the sum of the first-level loss and the second-level loss.

[0081] Example 2

[0082] In one or more embodiments, a hierarchical text classification system based on differentiated prompt characters is disclosed, including a template design module, a category differentiation module, a template filling module, and a category prediction module:

[0083] The template design module is configured to: design a hierarchical prediction prompt template with multiple pre-defined placeholders, wherein the placeholders include: text to be classified, first-level prompt character, first-level category mask, second-level prompt character, and second-level category mask;

[0084] The category differentiation module is configured to extract and aggregate semantic features from the first-level category set and the second-level category set respectively, to obtain differentiated prompt characters for the first and second levels.

[0085] The template filling module is configured to fill the text to be classified and the differential prompt characters of the first and second levels into the corresponding placeholders of the hierarchical prediction prompt template to obtain the sequence to be predicted;

[0086] The category prediction module is configured to input the sequence to be predicted into the pre-trained language model PLM, predict the first-level category mask and the second-level category mask, and obtain the final prediction result.

[0087] Example 3

[0088] The purpose of this embodiment is to provide a computer-readable storage medium.

[0089] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in a hierarchical text classification method based on differentiated prompt characters as described in Embodiment 1 of this disclosure.

[0090] Example 4

[0091] The purpose of this embodiment is to provide an electronic device.

[0092] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a hierarchical text classification method based on differentiated prompt characters as described in Embodiment 1 of this disclosure.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / mask> < / mask> < / masko> < / mask> < / t1> < / mask> < / t0> < / x> < / mask> < / t1> < / mask> < / t0> < / t0> < / mask0> < / t0> < / t0> < / mask0> < / t0> < / mask0> < / t1> < / mask0> < / t0> < / t0> < / mask> < / t1> < / mask> < / t0> < / t0> < / t0> < / mask0> < / t1> < / mask0> < / t0>

Claims

1. A hierarchical text classification method based on differentiated prompt characters, characterized in that, include: Design a hierarchical prediction prompt template with multiple pre-defined placeholders, wherein the placeholders include: text to be classified, first-level prompt character, first-level category mask, second-level prompt character, and second-level category mask; Semantic features are extracted and aggregated from the first-level category set and the second-level category set respectively to obtain differentiated prompt characters for the first and second levels; The semantic feature extraction and aggregation process involves the following steps: Each category in the category set is input into the pre-trained language model's embedding mechanism and a custom linear layer to generate an intermediate vector for each category; Hierarchical feature extraction and aggregation are performed on the intermediate vector to obtain the aggregated feature vector; The custom linear layer is a linear transformation that normalizes the regression values ​​of the categories; Fill the corresponding placeholders in the hierarchical prediction prompt template with the text to be classified, the differentiated prompt characters of the first level and the second level, to obtain the sequence to be predicted; The sequence to be predicted is input into a pre-trained language model to predict the first-level class mask and the second-level class mask, thus obtaining the final prediction result.

2. The hierarchical text classification method based on differentiated prompt characters as described in claim 1, characterized in that, The hierarchical prediction prompt template is < _ >< 0>< 0>< 1>< 1>, where, < 0>、< 1> represents the first-level category mask and the second-level category mask, respectively; < 0>、< 1> represents the first-level prompt character and the second-level prompt character, respectively. _ This indicates the text to be categorized.

3. The hierarchical text classification method based on differentiated prompt characters as described in claim 1, characterized in that, TextCNN is used to extract and aggregate semantic features from the first-level and second-level category sets respectively, resulting in differentiated prompt characters for the first and second levels, which are then used as the hierarchical prediction prompt template. 0>、< The fill value for 1>.

4. The hierarchical text classification method based on differentiated prompt characters as described in claim 1, characterized in that, The pre-trained language model is used to introduce the prediction sequence to the model, and the result tensor is obtained through the MLM Head.

5. The hierarchical text classification method based on differentiated prompt characters as described in claim 4, characterized in that, The final prediction result is obtained by selecting the regression values ​​of the first-level category mask and the second-level category mask from the result tensor, obtaining the probability of each category in the regression value, then synthesizing multiple probabilities through the linear layer of the corresponding category, and finally finding the index of the maximum value to obtain the first-level category and the second-level category.

6. A hierarchical text classification system based on differentiated prompt characters, characterized in that, It includes a template design module, a category differentiation module, a template filling module, and a category prediction module: The template design module is configured to: design a hierarchical prediction prompt template with multiple pre-defined placeholders, wherein the placeholders include: text to be classified, first-level prompt character, first-level category mask, second-level prompt character, and second-level category mask; The category differentiation module is configured to extract and aggregate semantic features from the first-level category set and the second-level category set respectively, to obtain differentiated prompt characters for the first and second levels. The semantic feature extraction and aggregation process involves the following steps: Each category in the category set is input into the pre-trained language model's embedding mechanism and a custom linear layer to generate an intermediate vector for each category; Hierarchical feature extraction and aggregation are performed on the intermediate vector to obtain the aggregated feature vector; The custom linear layer is a linear transformation that normalizes the regression values ​​of the categories; The template filling module is configured to fill the text to be classified and the differential prompt characters of the first and second levels into the corresponding placeholders of the hierarchical prediction prompt template to obtain the sequence to be predicted; The category prediction module is configured to: input the sequence to be predicted into the pre-trained language model, predict the first-level category mask and the second-level category mask, and obtain the final prediction result.

7. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-5.

8. A storage medium, characterized in that, The computer-readable instructions are stored non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-5 are executed.

Images