Theme classification method and system for multi-label text, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a theme classification method and system for a multi-label text, electronic equipment and a storage medium, and belongs to the field of theme classification. The method comprises the steps of obtaining a to-be-predicted multi-label text; the to-be-predicted multi-label text is input into a pre-constructed text classification model, a subject classification result of the to-be-predicted multi-label text is output, and the text classification model is constructed on the basis of a RoBerta-base algorithm in combination with the classification layer. According to the technical scheme, the improved text classification model is used, the model can pay more attention to a small number of samples, the influence of poor model effect caused by data long tail distribution is reduced, and the model recall rate is increased. In the subject classification task, the situation that the model effect is poor due to data imbalance can be effectively relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of topic classification, and in particular to a topic classification method, system, electronic device and storage medium for multi-label text. Background Art

[0002] Topic classification is a task that classifies text data according to topic categories. There are many methods to achieve topic classification, including: statistical learning methods such as naive Bayes and decision trees, as well as some deep learning methods. In the process of topic classification, there are two more difficult problems to solve: 1. Multi-label classification. The same document may involve multiple topics, and the coupling between some topics is high. 2. Long-tail distribution. This problem is mainly reflected in the data, that is, the data of different topics is extremely unbalanced.

[0003] Deep long-tail learning is a very challenging task in visual recognition, and it also often appears in natural language processing tasks. Long-tail distribution refers to a small number of categories in a data set that account for a large proportion of the total data set, namely the so-called "head" labels, while most categories have fewer samples and are concentrated in the "tail". There are many ways to solve the long-tail distribution problem, including sampling methods, mutual information methods, etc. There are many sampling methods, including oversampling by copying existing samples or synthesizing new data and reducing undersampling of most samples. Sampling is the most direct way to deal with data imbalance problems, but there are also many problems, including feature loss, overfitting, increased noise impact, and unsuitability for multi-label processing.

[0004] Sampling can solve the problem of sample imbalance to a certain extent, but for multi-label classification tasks, there may be a high degree of coupling between labels. Blind sampling not only fails to achieve the optimization effect, but may also cause greater impact. For example, news about some technology companies releasing new products involves the theme of technological innovation and contains information about the new products released. In addition, the news also mentions that the stock price rose due to the release of new products, involving both technology and financial themes. When there are multiple documents with both technological innovation themes and financial stock themes, directly sampling the financial stock theme will also affect the data of the technological innovation theme. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a topic classification method, system, electronic device and storage medium for multi-label text, which are used to fully or at least partially solve the technical problems existing in the above-mentioned prior art.

[0006] In order to achieve the above object, an embodiment of the present invention provides a topic classification method for multi-label text, comprising: Get the multi-label text to be predicted; The multi-label text to be predicted is input into a pre-built text classification model, and a topic classification result of the multi-label text to be predicted is output, wherein the text classification model is built based on the RoBerta-base algorithm and combined with a classification layer.

[0007] Optionally, the topic classification method for the multi-label text further includes: After obtaining the multi-label text to be predicted, the document to be predicted is segmented according to a preset length, and the segmented segments are subjected to special symbols removal and unitization to obtain the target multi-label text data to be predicted.

[0008] Optionally, the process of constructing the text classification model includes: Select the structure of the Roberta-base model as the basic architecture of the text classification model, and load the Roberta-base model through transformers; After encoding by the Roberta-base model, a fully connected layer is added to adjust the vector dimension; Add multiple independent binary classifiers after the fully connected layer for classification; Define a weighted loss function as the loss function for the text classification model.

[0009] Optionally, the loss function of the text classification model is as follows: ; In the formula, Used to correct the cross entropy loss generated in different situations. is the true label, For the prediction results.

[0010] Optional, ; In the formula, are the thresholds of different classifiers, is the true label, For the prediction result, K represents a real number.

[0011] Optionally, the multi-label text to be predicted is input into a pre-built text classification model, and a topic classification result of the multi-label text to be predicted is output, including: The multi-label text to be predicted is segmented into documents, special symbols are removed, and after unitization, the multi-label text to be predicted is converted into an input format that conforms to the text classification model; The multi-label text to be predicted that meets the input format of the text classification model is input into the Roberta-base model, and the embedding vector is output. The embedding vector is input into the fully connected layer, so that the embedding vector dimension is mapped to a low dimension to make it consistent with the classification category dimension, and then classified by multiple binary classifiers, where each classifier outputs a classification category; Map the results of multiple binary classifiers to determine the topic of the multi-label text to be predicted.

[0012] On the other hand, the present invention also provides a topic classification system for multi-label text, comprising: An acquisition unit, used to acquire multi-label text to be predicted; The classification unit is used to input the multi-label text to be predicted into a pre-built text classification model, and output a topic classification result of the multi-label text to be predicted, wherein the text classification model is constructed based on the RoBerta-base algorithm and combined with a classification layer.

[0013] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for topic classification of multi-label text when executing the program.

[0014] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for topic classification of multi-label texts.

[0015] Through the above technical solutions, using the improved text classification model, the model can focus more on a small number of samples, reduce the impact of poor model performance caused by long-tail data distribution, and improve the model recall rate. In the topic classification task, it can effectively alleviate the situation of poor model performance caused by data imbalance.

[0016] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 It is an implementation flow chart of a topic classification method for multi-label text provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a process of constructing a text classification model provided by an embodiment of the present invention; Figure 3 It is a flowchart for theme classification using a text classification model provided by an embodiment of the present invention; Figure 4 It is a detailed implementation flowchart of a theme classification method for multi-label text provided by an embodiment of the present invention; Figure 5 It is a schematic structural diagram of a theme classification system for multi-label text provided by an embodiment of the present invention. Detailed implementation manner

[0018] The following will describe in detail the specific implementation manners of the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention.

[0019] Refer to Figure 1 As shown, it is an implementation flowchart of a theme classification method for multi-label text provided by an embodiment of the present invention, including the following execution steps: Step 100: Obtain the multi-label text to be predicted.

[0020] In some implementation manners, after obtaining the multi-label text to be predicted, the multi-label text to be predicted is segmented according to a preset length, and special symbols are removed and unitized for the segmented segments to obtain the target multi-label text data to be predicted.

[0021] In a specific implementation manner, 1) Document segmentation: For overly long documents, perform length segmentation, intercept the content with a length of N at the front and back of the document, and N generally takes a value of 512 and can be adjusted according to different situations. 2) Remove special symbols: Remove long special symbols through regular expressions, that is, if there is a long text segment (with a length exceeding M, generally taking a value of 6 and can be adjusted according to different situations) and consisting only of special symbols or alphanumeric characters (including: ~!@#$%^&*()_+,. / >< / ’;《》【】[]{}a-zA-Z), replace it with a blank string, that is, directly delete it. 3) Tokenization: Represent the original text as smaller units (tokens), and these tokens can be mapped to numbers and vectors and applied to downstream NLP tasks, that is, text theme classification, and can be specifically implemented through AutoTokenizer in transformers.

[0022] Step 101: Input the multi-label text to be predicted into a pre-constructed text classification model, and output the theme classification result of the multi-label text to be predicted, where the text classification model is constructed based on the RoBerta-base algorithm and combined with a classification layer.

[0023] In some embodiments, the RoBerta pre-trained model is selected. As a robustly optimized BERT method, RoBerta can show good results in the Chinese field. This is because the model has made the following improvements based on BERT: 1) Dynamic masking: During the training process, each sentence will be randomly masked with multiple words, while BERT only masks 15%, which enables the model to learn more complex language patterns. 2) Training batches: A larger batch is used for training, which improves the training efficiency and performance of the model. 3) Training scale: A larger model parameter amount is used, and the training time is also increased, which enables the model to learn richer language knowledge. 4) Not applicable to the Next Sentence Prediction (NSP) task: Roberta removes the NSP task in BERT, reducing the complexity. Studies have shown that this task does not contribute significantly to model performance. The specific usage of Roberta is as follows: 1) First download the Roberta-base model; 2) Load the pre-trained Roberta-base model through transformers. 3) Construct the classification model structure. The classification model consists of Roberta-base and classification layer. The given text features, namely token embedding, are obtained through Roberta-base. Since downstream classification tasks are required, the token embedding of CLS is taken. After being encoded by Roberta-base, a fully connected layer is added to adjust the vector dimension. Finally, N independent binary classifiers are added after the fully connected layer for classification. 4) Fine-tune the text topic classification data. The loss function used in the fine-tuning process is the loss function implementation defined in this application.

[0024] In some embodiments, see Figure 2 As shown, the construction process of the text classification model includes the following steps: S1: Select the structure of the Roberta-base model as the basic architecture of the text classification model, and load the Roberta-base model through transformers; S2: Add a fully connected layer after encoding by the Roberta-base model to adjust the vector dimension; S3: Add multiple independent binary classifiers after the fully connected layer for classification; S4: Define the weighted loss function as the loss function of the text classification model.

[0025] By using the improved loss function, the model can focus more on a small number of samples during model training, reduce the impact of poor model performance caused by long-tail distribution of data, and improve model recall. In addition, since the function is differentiable, during model training, for samples that are correctly classified, the model can still maintain the correct classification of the sample. In topic classification tasks, it can effectively alleviate the situation of poor model performance caused by data imbalance.

[0026] Specifically, the loss function of the text classification model is as follows: ; In the formula, Used to correct the cross entropy loss generated in different situations. is the true label, For the prediction results.

[0027] in, ; In the formula, are the thresholds of different classifiers, is the true label, To predict the results, Represents a real number.

[0028] when , , at this time the loss approaches 0, when , , keep the cross entropy loss, when hour, , the cross entropy loss is the original loss During model training, when the number of negative samples is much larger than the number of positive samples, the model will be more inclined to negative samples. The improved loss function can make the negative samples produce smaller losses and the positive samples produce larger losses. For example, when the number of negative samples is much larger than the number of positive samples, the model will be more inclined to judge the samples as negative samples with a larger number of samples. If there are 990 negative samples in 1000 samples, if all samples are predicted as negative samples, the accuracy rate is as high as 99%. Using the improved loss function, in this case, the accuracy rate of negative samples is 2. is very small, while the positive samples Keeping it constant will force the model to start focusing on the positive samples.

[0029] In some implementations, the following optimization strategies are configured: 1) Setting evaluation indicators: Select precision, recall and accuracy as the evaluation indicators of the final result. The training process is evaluated by the validation set loss and accuracy. 2) Define hyperparameters: including Epochs (size is defined according to the size of the training set), Batch size (Based on the definition of the data set and hardware device, 16, 32, or 64 are generally selected), learning rate (select 4e-5), Hidden size (Based on the definition of the pre-trained model, the Boberta-base model is selected, so the hidden layer is 1024); 3) Loss function: weighted loss function is used; 4) Optimizer: Adam optimizer is generally used.

[0030] In some embodiments, the text classification model training evaluation is performed: the number of batches is set to the number of training set samples / batch_size according to the number of training samples, and the validation set data is evaluated by calculating the loss of the current stage model on the validation set data and predicting the accuracy and recall of the validation set. If the current loss is less than the loss of the model in the previous stage, the model is saved. The subsequent steps are the same. If the loss of the current stage is less than the loss of the model in the previous stage, the model is saved, otherwise, it is not saved. If the validation set loss does not decrease after more than 1000 batch iterations, the training is terminated.

[0031] In some embodiments, see Figure 3 As shown, when executing step 101, the following steps may be specifically performed: S1010: segment the multi-label text to be predicted into documents, remove special symbols, and convert the multi-label text to be predicted into an input format that conforms to the text classification model after unitization.

[0032] S1011: Input the multi-label text to be predicted that conforms to the input format of the text classification model into the Roberta-base model, output the embedding vector, and input the embedding vector into the fully connected layer to map the embedding vector dimension to a low dimension so that it is consistent with the classification category dimension, and then classify it through multiple binary classifiers, where each classifier outputs a corresponding classification category.

[0033] S1012: Map the output results of multiple binary classifiers to determine the topic of the multi-label text to be predicted.

[0034] In a specific implementation, the text prediction process is as follows: 1) Input the text to be predicted. 2) The text is preprocessed by a module to remove special symbols and determine whether the text length is too long. 3) After Tokenization, the text is converted into an input format acceptable to the model. 4) Use the trained model for prediction: pass the output of the previous step into the BoBerta model, and after obtaining the model output, that is, token embedding, take the CLS token embedding, input it into the fully connected layer, map the vector dimension to a low dimension (consistent with the classification category), and then classify it through multiple classifiers. Each classifier corresponds to a classification category, and each classifier will output 0 or 1, that is, whether it belongs to this category. If the classification result is 1, it belongs to this category. 5) Map according to the final result of the classifier and return the topic of the document.

[0035] In some embodiments, see Figure 4 As shown, it is a detailed implementation flow chart of a topic classification method of a multi-label text provided by an embodiment of the present invention, including the following execution steps: S400: Obtain multi-label text to be predicted.

[0036] S401: The document to be predicted multi-label text is segmented according to a preset length, and special symbols are removed from the segmented segments and unitized to obtain target multi-label text data to be predicted.

[0037] S402: Select the structure of the Roberta-base model as the basic architecture of the text classification model, and load the Roberta-base model through transformers.

[0038] S403: After being encoded by the Roberta-base model, a fully connected layer is added to adjust the vector dimension.

[0039] S404: Add multiple independent binary classifiers after the fully connected layer for classification.

[0040] S405: Define a weighted loss function as the loss function of the text classification model.

[0041] S406: Input the multi-label text to be predicted that conforms to the input format of the text classification model into the Roberta-base model, output the embedding vector, and input the embedding vector into the fully connected layer, so that the embedding vector dimension is mapped to a low dimension so that it is consistent with the classification category dimension, and then classified by multiple binary classifiers, wherein each classifier outputs a corresponding classification category.

[0042] S407: Map the output results of multiple binary classifiers to determine the topic of the multi-label text to be predicted.

[0043] By adopting the weighted loss function, we can solve the problem of poor model performance due to the long tail phenomenon of data in multi-label text classification tasks to a certain extent, improve the recall rate of the model, and the improved loss function can effectively improve the classification effect of topic classification.

[0044] See also Figure 5 FIG. 1 is a schematic diagram of a structure of a topic classification system for multi-label text provided by an embodiment of the present invention, including: An acquisition unit 50 is used to acquire a multi-label text to be predicted; The classification unit 51 is used to input the multi-label text to be predicted into a pre-built text classification model, and output a topic classification result of the multi-label text to be predicted, wherein the text classification model is constructed based on the RoBerta-base algorithm and combined with the classification layer.

[0045] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the topic classification method for multi-label text described in any one of the above embodiments are implemented.

[0046] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the topic classification method for multi-label text described in any of the above embodiments.

[0047] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0048] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0049] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0051] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0052] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0053] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0054] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0055] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A topic classification method for multi-label text, characterized in that: include: Get the multi-label text to be predicted; The multi-label text to be predicted is input into a pre-built text classification model, and a topic classification result of the multi-label text to be predicted is output, wherein the text classification model is built based on the RoBerta-base algorithm and combined with a classification layer.

2. The topic classification method of multi-label text according to claim 1 is characterized in that: The topic classification method of the multi-label text also includes: After obtaining the multi-label text to be predicted, the document to be predicted is segmented according to a preset length, and the segmented segments are subjected to special symbols removal and unitization to obtain the target multi-label text data to be predicted.

3. The topic classification method of multi-label text according to claim 1 is characterized in that: The construction process of the text classification model includes: Select the structure of the Roberta-base model as the basic architecture of the text classification model, and load the Roberta-base model through transformers; After encoding by the Roberta-base model, a fully connected layer is added to adjust the vector dimension; Add multiple independent binary classifiers after the fully connected layer for classification; Define a weighted loss function as the loss function for the text classification model.

4. The topic classification method of multi-label text according to claim 3 is characterized in that: The loss function of the text classification model is as follows: ； In the formula, Used to correct the cross entropy loss generated in different situations. is the true label, For the prediction results.

5. The topic classification method of multi-label text according to claim 4 is characterized in that: ； In the formula, are the thresholds of different classifiers, is the true label, For the prediction result, K represents a real number.

6. The topic classification method of multi-label text according to claim 1 is characterized in that: Input the multi-label text to be predicted into a pre-built text classification model, and output the topic classification result of the multi-label text to be predicted, including: The multi-label text to be predicted is segmented into documents, special symbols are removed, and after unitization, the multi-label text to be predicted is converted into an input format that conforms to the text classification model; The multi-label text to be predicted that meets the input format of the text classification model is input into the Roberta-base model, and the embedding vector is output. The embedding vector is input into the fully connected layer, so that the embedding vector dimension is mapped to a low dimension to make it consistent with the classification category dimension, and then classified by multiple binary classifiers, where each classifier outputs a classification category; Map the results of multiple binary classifiers to determine the topic of the multi-label text to be predicted.

7. A topic classification system for multi-label text, characterized in that: include: An acquisition unit, used to acquire multi-label text to be predicted; The classification unit is used to input the multi-label text to be predicted into a pre-built text classification model, and output a topic classification result of the multi-label text to be predicted, wherein the text classification model is constructed based on the RoBerta-base algorithm and combined with a classification layer.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the topic classification method for multi-label text as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the topic classification method for multi-label text as described in any one of claims 1 to 6 are implemented.

Images