Text classification and model training method, system, device and storage medium

Abstract

The present invention provides a text classification and model training method, system, device and storage medium, wherein the text classification model training method comprises: encoding text sample data on an encoder to obtain text feature information; extracting target class text features corresponding to a first text classification label from the text feature information on a first sub-model and predicting a first text classification result corresponding to the first text classification label; splicing the text feature information and the target class text features on a second sub-model and predicting a second text classification result corresponding to a second text classification label based on the spliced ​​text features; obtaining a target loss function of a text classification model based on a first loss function of the first text classification result and a second loss function of the second text classification result, and training the text classification model using the target loss function until a training stop condition is reached. The model training method of the present invention enables knowledge to be shared between tasks and improves the results of each text classification task.

Description

Background Art

[0002] Natural Language Processing (NLP) is a core tool for text analysis and mining, which aims to process text efficiently. Among them, classification is one of the most common tasks in natural language processing.

[0003] Text classification (TC), also known as automatic text classification, refers to the process of mapping text containing information to a certain category or several categories of topics given in advance. Through text classification, human resources are saved for various intelligent scenarios. Therefore, the text classification problem is a very classic problem in the field of natural language processing. Summary of the invention

[0004] In view of the problems in the prior art, the purpose of the present invention is to provide a text classification and model training method, system, device and storage medium, which overcomes the difficulties of the prior art and can improve the accuracy of the model results.

[0005] An embodiment of the present invention provides a text classification model training method, wherein the text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder;

[0006] Text classification model training methods include:

[0007] Encoding the text sample data on the encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different text classification tasks respectively;

[0008] On the first sub-model, extracting target class text features corresponding to the first text classification label from the text feature information, and predicting a first text classification result corresponding to the first text classification label based on the target class text features;

[0009] On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and a second text classification result corresponding to the second text classification label is predicted based on the concatenated text feature;

[0010] A target loss function of the text classification model is obtained according to a first loss function between a first text classification result and a first text classification label, and a second loss function between a second text classification result and a second text classification label, and the text classification model is trained using the target loss function until a training stop condition is reached.

[0011] Optionally, the text classification model also includes a tokenizer;

[0012] Before encoding the text sample data on the encoder, the text classification model training method also includes:

[0013] A target text sample is obtained, and word segmentation and sentence segmentation operations are performed on the target text sample on the word segmenter. A digital sequence is obtained corresponding to the word segmentation operation, and a fragmentation sequence containing the digital sequence is obtained corresponding to the sentence segmentation operation. Text sample data is obtained according to the fragmentation sequence.

[0014] Optionally, obtaining a target text sample includes:

[0015] Obtaining an original text sample, segmenting the original text sample based on a sentence end identifier to obtain a sub-text sample;

[0016] When the text length of the sub-text sample is not shorter than the preset target text length, the sub-text sample is segmented to obtain a target text sample whose text length is shorter than the target text length.

[0017] Optionally, before segmenting the original text sample based on the sentence end identifier to obtain the sub-text sample, the text classification model training method further includes:

[0018] Perform data preprocessing on the original text samples.

[0019] Optionally, the encoder is a transformer-based bidirectional encoder;

[0020] Encode the text sample data on the encoder to obtain text feature information, including:

[0021] The text sample data is input into the transformer-based bidirectional encoder to obtain text feature information.

[0022] Optionally, the first sub-model includes a cascaded self-attention layer and at least one first fully connected layer;

[0023] On the first sub-model, the target class text features corresponding to the first text classification label are extracted from the text feature information, including:

[0024] The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text feature corresponding to the first text classification label.

[0025] Optionally, the first sub-model further includes a cascaded second fully connected layer and a normalized index;

[0026] Predicting a first text classification result corresponding to a first text classification label according to the target class text feature includes:

[0027] The target class text features are processed on the second fully connected layer and the normalized index to obtain a first text classification result corresponding to the first text classification label.

[0028] Optionally, the second sub-model includes a cascaded third fully connected layer and an activation function;

[0029] Predicting a second text classification result corresponding to the second text classification label according to the concatenated text features includes:

[0030] The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification label.

[0031] Optionally, when the text sample data includes a digital sequence, predicting a first text classification result corresponding to a first text classification label according to the target class text feature includes:

[0032] A target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification label is predicted for the target feature vector on the first sub-model.

[0033] Optionally, when the text sample data includes a digital sequence, predicting a second text classification result corresponding to a second text classification label based on the concatenated text features on the second sub-model includes:

[0034] A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second text classification label is predicted for the second target feature vector on the second sub-model.

[0035] An embodiment of the present invention further provides a text classification method based on a text classification model, wherein the text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder;

[0036] Text classification methods include:

[0037] Encode the target text data on the encoder to obtain text feature information;

[0038] On the first sub-model, extracting target class text features corresponding to the first text classification task from the text feature information, and predicting a first text classification result corresponding to the first text classification task based on the target class text features;

[0039] In the second sub-model, the text feature information and the target class text features are concatenated to obtain concatenated text features, and the second text classification result corresponding to the second text classification task is predicted based on the concatenated text features.

[0040] Optionally, the text classification model further includes a word segmenter;

[0041] Text classification methods also include:

[0042] Before encoding the target text data on the encoder, the text to be processed is obtained, and the text to be processed is subjected to word segmentation and sentence segmentation operations on the word segmenter. A numerical sequence is obtained corresponding to the word segmentation operation, and a fragmentation sequence containing the numerical sequence is obtained corresponding to the sentence segmentation operation. The target text data is obtained according to the fragmentation sequence.

[0043] Optionally, obtain the text to be processed, including:

[0044] Obtaining an original text, segmenting the original text based on sentence end identifiers to obtain sub-texts;

[0045] When the text length of the subtext is not shorter than the preset target text length, the subtext is segmented to obtain a to-be-processed text whose text length is shorter than the target text length.

[0046] Optionally, before segmenting the original text based on the sentence end identifier to obtain sub-texts, the text classification method further includes:

[0047] Perform data preprocessing on the original text.

[0048] Optionally, the encoder is a transformer-based bidirectional encoder;

[0049] The target text data is encoded on the encoder to obtain text feature information, including:

[0050] The target text data is input into the transformer-based bidirectional encoder to obtain text feature information.

[0051] Optionally, the first sub-model includes a cascaded self-attention layer and at least one first fully connected layer;

[0052] In the first sub-model, the target class text features corresponding to the first text classification task are extracted from the text feature information, including:

[0053] The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text features corresponding to the first text classification task.

[0054] Optionally, the first sub-model further includes a cascaded second fully connected layer and a normalized index;

[0055] Predicting a first text classification result corresponding to the first text classification task according to the target class text feature includes:

[0056] The target class text features are processed on the second fully connected layer and the normalized index to obtain the first text classification result corresponding to the first text classification task.

[0057] Optionally, the second sub-model includes a cascaded third fully connected layer and an activation function;

[0058] Predicting the second text classification result corresponding to the second text classification task according to the concatenated text features includes:

[0059] The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification task.

[0060] Optionally, when the target text data includes a digital sequence, predicting a first text classification result corresponding to the first text classification task according to the target class text feature includes:

[0061] A first target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification task is predicted for the first target feature vector on the first sub-model.

[0062] Optionally, when the target text data includes a digital sequence, predicting a second text classification result corresponding to the second text classification task according to the concatenated text feature includes:

[0063] A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second text classification is predicted for the second target feature vector on the second sub-model.

[0064] Optionally, the text classification method further includes:

[0065] When the target text data includes at least two sub-text data, and the second text classification label sets contained in the second text classification results of the two adjacent sub-texts have an intersection, the two adjacent sub-texts are merged, and the union of the second text classification label sets of the two adjacent sub-texts is taken, and the union is used as the second text classification result of the merged sub-text.

[0066] An embodiment of the present invention further provides a text classification model training system, wherein the text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder;

[0067] The text classification model training system includes:

[0068] A first encoding module encodes the text sample data on an encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different text classification tasks respectively;

[0069] A first text classification module, on the first sub-model, extracts target class text features corresponding to the first text classification label from the text feature information, and predicts a first text classification result corresponding to the first text classification label based on the target class text features;

[0070] The second text classification module, on the second sub-model, concatenates the text feature information and the target class text feature to obtain a concatenated text feature, and predicts a second text classification result corresponding to the second text classification label based on the concatenated text feature;

[0071] The training module obtains a target loss function of the text classification model according to a first loss function between a first text classification result and a first text classification label, and a second loss function between a second text classification result and a second text classification label, and trains the text classification model using the target loss function until a training stop condition is reached.

[0072] An embodiment of the present invention further provides a text classification system based on a text classification model, wherein the text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder;

[0073] The text classification system includes:

[0074] The second encoding module encodes the target text data on the encoder to obtain text feature information;

[0075] A first text classification module, on the first sub-model, extracts target class text features corresponding to the first text classification task from the text feature information, and predicts a first text classification result corresponding to the first text classification task based on the target class text features;

[0076] The second text classification module, on the second sub-model, concatenates the text feature information and the target class text feature to obtain the concatenated text feature, and predicts the second text classification result corresponding to the second text classification task based on the concatenated text feature.

[0077] An embodiment of the present invention further provides an electronic device, including:

[0078] processor;

[0079] a memory in which executable instructions for the processor are stored;

[0080] The processor is configured to execute the steps of the above-mentioned text classification model training method or the text classification method based on the text classification model by executing executable instructions.

[0081] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned text classification model training method or the text classification method based on the text classification model.

[0082] The object of the present invention is to provide a text classification and model training method, system, device and storage medium. The text classification model includes an encoder and a first sub-model and a second sub-model of the common encoder. The text classification model training method includes: encoding text sample data on the encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label respectively correspond to different types of text classification tasks; on the first sub-model, extracting a target class text feature corresponding to the first text classification label from the text feature information, and predicting a first text classification result corresponding to the first text classification label according to the target class text feature; on the second sub-model, splicing the text feature information and the target class text feature to obtain a spliced ​​text feature, and predicting a second text classification result corresponding to the second text classification label according to the spliced ​​text feature; obtaining a target loss function of the text classification model according to a first loss function between the first text classification result and the first text classification label, and a second loss function between the second text classification result and the second text classification label, and training the text classification model using the target loss function until a training stop condition is reached.

[0083] The disclosed embodiment proposes a scheme for joint training of different types of text classification tasks. Among them, by splicing text feature information with target class text features, the target class text features learned by the first sub-model are shared with the second sub-model, and the second sub-model can learn more generalized features, reduce overfitting, and improve the accuracy of model results. At the same time, joint training using a multi-task learning architecture allows knowledge to be shared between tasks, improving the results of each text classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0085] Figure 1 This is one of the flowcharts of the text classification model training method provided by an embodiment of the present invention;

[0086] Figure 2 This is the second flowchart of the text classification model training method provided by the embodiment of the present invention;

[0087] Figure 3 is a schematic diagram of a data preprocessing method provided by an embodiment of the present invention;

[0088] Figure 4 This is one of the flow charts of the text classification method provided by the embodiment of the present invention;

[0089] Figure 5 This is the second flowchart of the text classification method provided by the embodiment of the present invention;

[0090] Figure 6 This is the third flowchart of the text classification method provided by the embodiment of the present invention;

[0091] Figure 7 is an architecture diagram of a text classification model training system provided by an embodiment of the present invention;

[0092] Figure 8 is an architecture diagram of a text classification system provided by an embodiment of the present invention;

[0093] Fig. 9 It is one of the module schematic diagrams of the text classification model training system provided by an embodiment of the present invention;

[0094] Fig.10 This is the second module schematic diagram of the text classification model training system provided by an embodiment of the present invention;

[0095] Fig.11 This is the third module schematic diagram of the text classification model training system provided by an embodiment of the present invention;

[0096] Fig.12 It is one of the module schematic diagrams of the text classification system based on the text classification model provided by the embodiment of the present invention;

[0097] Fig.13 This is the second module schematic diagram of the text classification system based on the text classification model provided by the embodiment of the present invention;

[0098] Fig.14 This is the third module schematic diagram of the text classification system based on the text classification model provided by the embodiment of the present invention;

[0099] Fig.15 This is the third module schematic diagram of the text classification system based on the text classification model provided by the embodiment of the present invention;

[0100] Fig.16 It is a schematic diagram of the operation of the electronic device of the present invention. DETAILED DESCRIPTION

[0101] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to make the present invention comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0102] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware forwarding modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0103] Traditional text classification methods usually classify texts through manual annotation, but the efficiency of text classification is low and cannot meet the classification requirements of massive text data. With the development of machine learning, some text classification methods based on deep learning models have been proposed.

[0104] In the related art, text sample data is usually used to train a model based on the goal of a single classification task, so that the trained model can be applied to the single classification task.

[0105] However, in relevant practice, a text may usually be interpreted in multiple ways, so it can be used for multiple different text classification tasks, and there is a certain correlation between these text classification tasks.

[0106] Therefore, the embodiment of the present disclosure proposes an improved text classification scheme, the inventive idea of ​​which is to use a multi-task learning architecture for joint training, to jointly train multiple different text classification tasks, so that knowledge can be shared between different text classification tasks, thereby improving the accuracy of each text classification task.

[0107] Figure 1 The flowchart of the text classification model training method provided by the embodiment of the present disclosure is a text classification system, wherein the text classification model includes an encoder, a first sub-model and a second sub-model. Figure 1 As shown, the text classification model training method includes the following steps:

[0108] Step 110: Encode the text sample data on the encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different text classification tasks respectively;

[0109] Step 120: on the first sub-model, extracting target class text features corresponding to the first text classification label from the text feature information, and predicting a first text classification result corresponding to the first text classification label according to the target class text features;

[0110] Step 130: On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and on the second sub-model, a second text classification result corresponding to the second text classification label is predicted according to the concatenated text feature;

[0111] Step 140: Obtain a target loss function of the text classification model based on a first loss function between the first text classification result and the first text classification label, and a second loss function between the second text classification result and the second text classification label, and train the text classification model using the target loss function until a training stop condition is reached.

[0112] In the disclosed embodiment, different types of text classification tasks are jointly trained. Specifically, by splicing text feature information with target class text features, the target class text features learned by the first sub-model are shared with the second sub-model, and the second sub-model can learn more generalized features, reduce overfitting, and improve the accuracy of model results. At the same time, a multi-task learning architecture is used for joint training to share knowledge between tasks and improve the results of each text classification task.

[0113] In one scenario, there is a certain correlation between different text classification tasks. This correlation is reflected in the fact that the second text classification label itself can reflect the characteristics of the first text classification label. In this case, sharing the target class text features learned by the first sub-model with the second sub-model can improve the text classification results of the second sub-model.

[0114] In the embodiment of the present disclosure, the text classification model also includes a word segmenter. Based on this, Figure 2 As shown, the embodiment of the present disclosure also provides another text classification model training method, which includes the following steps:

[0115] Step 210: Obtain a target text sample, perform word segmentation and sentence segmentation operations on the target text sample on a word segmenter, obtain a digital sequence corresponding to the word segmentation operation, obtain a fragmentation sequence including the digital sequence corresponding to the sentence segmentation operation, and obtain text sample data according to the fragmentation sequence;

[0116] Step 220: Encode the text sample data on the encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different text classification tasks respectively;

[0117] Step 230: on the first sub-model, extracting target class text features corresponding to the first text classification label from the text feature information, and predicting a first text classification result corresponding to the first text classification label according to the target class text features;

[0118] Step 240: On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and on the second sub-model, a second text classification result corresponding to the second text classification label is predicted based on the concatenated text feature;

[0119] Step 250: Obtain a target loss function of the text classification model based on a first loss function between the first text classification result and the first text classification label, and a second loss function between the second text classification result and the second text classification label, and train the text classification model using the target loss function until a training stop condition is reached.

[0120] In the disclosed embodiment, a word segmenter is used to digitally encode text elements, such as characters or words, in a target text sample, and each word in the target text sample can be converted into a digital sequence. The segmentation sequence is obtained by segmenting the target text sample, and each segment includes a digital sequence obtained by at least one text element. The final segmentation sequence is composed of a series of digital sequences in sequence.

[0121] The disclosed embodiment can automatically segment text samples into sentences, achieve the purpose of segmenting first and then extracting feature information, and significantly improve the efficiency of text classification processing.

[0122] In an optional embodiment of the present disclosure, the text classification model may not include a word segmenter. In this case, the target text sample may also be obtained by performing word segmentation and sentence segmentation operations in advance, without the need for performing word segmentation operations in real time.

[0123] In the embodiment of the present disclosure, obtaining a target text sample specifically includes the following steps:

[0124] Obtaining an original text sample, segmenting the original text sample based on a sentence end identifier to obtain a sub-text sample;

[0125] When the text length of the sub-text sample is not shorter than the preset target text length, the sub-text sample is segmented to obtain a target text sample whose text length is shorter than the target text length.

[0126] In this case, the segmented sub-texts may be annotated with the first text classification label and / or the second text classification label to be made into independent target text samples as input of the word segmenter.

[0127] In the disclosed embodiment, the sentence end identifier may be a period, a comma, a semicolon, an ellipsis, etc., or may be other identifiers marked in advance, which are not limited here.

[0128] The disclosed embodiment can realize segmenting a long text into sub-texts according to rules such as sentence end identifier and target text length, thereby improving model processing efficiency.

[0129] The target text length can be set according to the corresponding hyperparameter of the text classification model, and is not specifically limited. When segmenting the sub-text samples, the segmentation can be based on the sentence pause identifier, for example, when the sentence end identifier is a period, the sentence pause identifier is a comma, a colon, or a semicolon.

[0130] In other embodiments, when the text length of the original text sample is longer than the target text length, no segmentation operation is required.

[0131] In the embodiment of the present disclosure, before segmenting the original text sample based on the sentence end identifier to obtain the sub-text sample, the text classification model training method further includes:

[0132] Perform data preprocessing on the original text samples.

[0133] Among them, data preprocessing can refer to manipulating or deleting the data of the original text sample before using it to ensure or enhance performance.

[0134] Specifically, refer to Figure 3 ,Data preprocessing includes one or more of Emoji (visual emotional symbols used in wireless communication) replacement, floating point number replacement, list number replacement, English punctuation to Chinese punctuation and end symbol replacement.

[0135] Emoji replacement replaces Emojis that the model cannot parse with their corresponding parseable Chinese expressions. Floating point replacement replaces all positive and negative floating point numbers with a specific symbol. List number replacement replaces all list numbers with "." to represent sentence breaks. English punctuation to Chinese punctuation converts half-width English punctuation marks to Chinese punctuation marks. End symbol replacement replaces the symbol representing the end of a sentence with ".".

[0136] In another embodiment, data preprocessing may also include steps such as data cleaning, data integration, data reduction or data transformation, which are not specifically limited here.

[0137] In the disclosed embodiment, the encoder may use a pre-trained model based on a bidirectional encoder representation technology (BiDirectional Encoder Representation From Transformers, abbreviated as BERT). In this case, the text sample data is encoded on the encoder to obtain text feature information, including:

[0138] The text sample data is input into the transformer-based bidirectional encoder to obtain text feature information.

[0139] The BERT model trains deep bidirectional representations by combining bidirectional transformers in all layers. The BERT model combines the advantages of many natural language processing models and achieves excellent results in multiple natural language processing tasks.

[0140] Among them, the BERT is a pre-training model. Pre-training is a process of training a model using a large data set so that the model learns the common features in the data set. The purpose of pre-training is to provide high-quality model parameters for subsequent model training on a specific data set. Pre-training in the embodiments of the present disclosure refers to the process of training the BERT model using unlabeled training text.

[0141] The pre-trained model learns the common features of the text and has the ability to extract common text features. After that, it goes through the first sub-model and the second sub-model to focus on learning features for specific text classification tasks to obtain good model effects in corresponding scenarios.

[0142] In this case, the text sample data may include a mask sequence, which is obtained by adding a mask to the digital sequence of the corresponding text element.

[0143] The mask may be a preset special token, for example, the mask may be [MASK]. Since the text element to be predicted is replaced by a mask, the model's ability to express and generalize the semantics of text messages is increased, and on the other hand, the efficiency of the training process is improved.

[0144] In another embodiment of the present disclosure, the encoder may also use other unidirectional or bidirectional converters such as a converter.

[0145] In the embodiment of the present disclosure, the first sub-model includes a cascaded self-attention layer and at least one first fully connected layer. In this case, on the first sub-model, extracting the target class text feature corresponding to the first text classification label from the text feature information includes:

[0146] The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text feature corresponding to the first text classification label.

[0147] The self-attention mechanism breaks the structural constraints between the text elements of the input sequence, so that the self-attention of each text element can be remembered independently to increase the training speed. Therefore, the self-attention layer can focus on learning about the first text classification label.

[0148] Among them, the self-attention layer calculates the self-attention feature A based on the text feature information based on the following formula:

[0149]

[0150] Where Q = W Q H1, K = W K H1, V = W V H1, d=hidden.

[0151] In the disclosed embodiment, the first sub-model may include a multi-level first fully connected layer, and an activation function is set between two adjacent first fully connected layers. The fully connected layer is a layer in which each node is connected to all nodes of the previous layer, so as to integrate the features extracted previously, thereby finally obtaining the target class text features.

[0152] Based on the above embodiment, the multi-level first fully connected layer includes a first-level first fully connected layer, a second-level first fully connected layer and a third-level first fully connected layer. The above A passes through the first-level first fully connected layer to obtain:

[0153] O1=Dense1(A)∈R b×max_sqe×hidden

[0154] After activating O1 with the sigmoid function, input the first fully connected layer of the second level to obtain:

[0155] O2=Dense2(O1)∈R b×max_sqe×2hidden

[0156] O2 is activated using the linear rectification ReLU function, then input into the first fully connected layer of the third level and activated using Sigmoid to obtain:

[0157] H2=Sigmoid(Dense3(O2))∈R b×max_sqe×hidden

[0158] The formula is as follows:

[0159] H2=Sigmoid(Dense3(ReLU(Dense2(Sigmoid(Dense1(A))))))).

[0160] In the disclosed embodiment, the first sub-model further includes a cascaded second fully connected layer and a normalized index. In this case, predicting the first text classification result corresponding to the first text classification label according to the target class text feature includes:

[0161] The target class text features are processed on the second fully connected layer and the normalized index to obtain a first text classification result corresponding to the first text classification label.

[0162] In this case, the self-attention layer, at least one first fully connected layer, the second fully connected layer and the normalized index are cascaded in sequence.

[0163] The first text classification result is a text classification result corresponding to the first text classification task, and there is an error between the first text classification result and the text type marked by the first text classification label. The error can be calculated using the corresponding first loss function.

[0164] In the disclosed embodiment, the second sub-model includes a cascaded third fully connected layer and an activation function. In this case, predicting the second text classification result corresponding to the second text classification label according to the concatenated text features includes:

[0165] The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification label.

[0166] In the embodiment of the present disclosure, the activation function selects a sigmoid function or other feasible functions.

[0167] The second text classification result is a text classification result corresponding to the second text classification task. There is an error between the second text classification result and the text type marked by the second text classification label. The error can be calculated using the corresponding second loss function.

[0168] In an embodiment of the present disclosure, when the text sample data includes a digital sequence, predicting a first text classification result corresponding to a first text classification label according to a target class text feature includes:

[0169] A first target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification label is predicted for the first target feature vector on the first sub-model.

[0170] After the encoder and the first sub-model, the target feature vector corresponding to the first digit sequence in the target class text feature actually contains the overall context information, that is, the text information of all subsequent digit sequences. Therefore, the second target feature vector corresponding to the first digit sequence can be used to predict the first text classification result.

[0171] For example, the digital sequence corresponding to "hello" is 123 4567, where 123 is the digital sequence of "you", 4567 is the digital sequence of "good", and "123" is the first digital sequence.

[0172] In an embodiment of the present disclosure, when the text sample data includes a digital sequence, predicting a second text classification result corresponding to a second text classification label based on the concatenated text features on the second sub-model includes:

[0173] A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second text classification label is predicted for the second target feature vector on the second sub-model.

[0174] In the embodiment of the present disclosure, the first loss function and the second loss function select a logarithmic cross entropy loss function or other types of loss functions.

[0175] In an embodiment of the present disclosure, obtaining a target loss function of a text classification model according to a first loss function between a first text classification result and a first text classification label, and a second loss function between a second text classification result and a second text classification label may include:

[0176] Based on the following formula, according to the first loss function L emo and the second loss function L e Calculate the target loss function L:

[0177] L=p w L emo +(1-p w )L e

[0178] Among them, w is a trainable parameter, p w =sigmoid(w)∈(0,1).

[0179] The disclosed embodiment introduces a trainable parameter w and calculates p through a sigmoid activation function w , the first loss function and the second loss function are integrated.

[0180] When the final loss function L is obtained, the loss calculated by the target loss function is transferred backwards to update all trainable model parameters in the text classification model. After several cycles of training, w is kept unchanged and the model is continued to be trained until the model converges and the training is stopped.

[0181] Among them, the training stop condition may be that the model accuracy reaches a preset requirement, the number of training rounds reaches a preset number of rounds, or the training duration reaches a preset duration, etc., which is not limited in the embodiments of the present application.

[0182] Figure 4 The flowchart of the text classification method provided by the embodiment of the present disclosure is a text classification system, wherein the text classification model includes an encoder, a first sub-model and a second sub-model. Figure 4 As shown, the text classification method includes the following steps:

[0183] Step 410: Encode the target text data on the encoder to obtain text feature information;

[0184] Step 420: extracting target text features corresponding to the first text classification task from the text feature information on the first sub-model, and predicting a first text classification result corresponding to the first text classification task based on the target text features;

[0185] Step 430: On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and a second text classification result corresponding to the second text classification task is predicted based on the concatenated text feature.

[0186] In the disclosed embodiment, by splicing the text feature information with the target class text features, the target class text features extracted by the first sub-model are shared with the second sub-model, and the second sub-model can use more generalized text features to predict the second text classification result, and the second text classification result has high accuracy. In addition, since the first sub-model and the second sub-model are cascaded together through the encoder, the text feature information extracted by the encoder has good feature generalization for both the first sub-model and the second sub-model, which to a certain extent makes the first text classification result and the second text classification result have good accuracy.

[0187] In the embodiment of the present disclosure, the text classification model is used Figure 1 or Figure 2 The text classification model is obtained by training using the training method shown, or it may be obtained using other training methods, which is not limited here.

[0188] In the embodiment of the present disclosure, the text classification model also includes a word segmenter. Based on this, Figure 5 As shown, the embodiment of the present disclosure also provides another text classification method, which specifically includes the following steps:

[0189] Step 510: obtaining a text to be processed, performing word segmentation and sentence segmentation operations on the text to be processed on a word segmenter, obtaining a digital sequence corresponding to the word segmentation operation, obtaining a fragmentation sequence including the digital sequence corresponding to the sentence segmentation operation, and obtaining target text data according to the fragmentation sequence;

[0190] Step 520: Encode the target text data on the encoder to obtain text feature information;

[0191] Step 530: extracting target text features corresponding to the first text classification task from the text feature information on the first sub-model, and predicting a first text classification result corresponding to the first text classification task based on the target text features;

[0192] Step 540: On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and a second text classification result corresponding to the second text classification task is predicted based on the concatenated text feature.

[0193] In the embodiment of the present disclosure, a word segmenter is used to digitally encode text elements, such as characters or words, in the text to be processed, and each text element in the text to be processed can be converted into a digital sequence. The segmentation sequence is obtained by segmenting the text to be processed, and the final segmentation sequence is composed of digital sequences corresponding to each text element in sequence.

[0194] The disclosed embodiment can automatically segment the text to be processed, so as to achieve the purpose of segmenting the text first and then extracting feature information, thereby significantly improving the efficiency of text classification processing.

[0195] In the embodiment of the present disclosure, obtaining the text to be processed includes:

[0196] Obtaining an original text, segmenting the original text based on sentence end identifiers to obtain sub-texts;

[0197] When the text length of the subtext is not shorter than the preset target text length, the subtext is segmented to obtain a to-be-processed text whose text length is shorter than the target text length.

[0198] When segmenting the subtext, the segmentation may be performed based on the sentence pause identifier. For example, when the sentence end identifier is a period, the sentence pause identifier is a comma, a colon, or a semicolon.

[0199] In other embodiments, when the text length of the original text is longer than the target text length, no segmentation operation is required.

[0200] The disclosed embodiment can segment a long text into sub-texts according to certain rules, which can improve the model processing efficiency.

[0201] In the embodiment of the present disclosure, before segmenting the original text based on the sentence end identifier to obtain the subtext, the text classification method further includes:

[0202] Perform data preprocessing on the original text.

[0203] Among them, data preprocessing can refer to manipulating or deleting the data of the original text before using it to ensure or enhance performance.

[0204] Specifically, data preprocessing includes one or more of Emoji (visual emotional symbols used in wireless communications) replacement, floating point number replacement, list number replacement, conversion of English punctuation to Chinese punctuation, and end symbol replacement.

[0205] Emoji replacement replaces Emojis that the model cannot parse with their corresponding parseable Chinese expressions. Floating point replacement replaces all positive and negative floating point numbers with a specific symbol. List number replacement replaces all list numbers with "." to represent sentence breaks. English punctuation to Chinese punctuation converts half-width English punctuation marks to Chinese punctuation marks. End symbol replacement replaces the symbol representing the end of a sentence with ".".

[0206] In another embodiment, data preprocessing may also include steps such as data cleaning, data integration, data reduction or data transformation, which are not specifically limited here.

[0207] In the disclosed embodiment, the encoder may use a pre-trained model of BERT, a bidirectional encoder representation technology based on a transformer. In this case, the target text data is encoded on the encoder to obtain text feature information, including:

[0208] The target text data is input into the transformer-based bidirectional encoder to obtain text feature information.

[0209] In the disclosed embodiments, the encoder may also use other unidirectional or bidirectional converters.

[0210] In the embodiment of the present disclosure, the first sub-model includes a cascaded self-attention layer and at least one first fully connected layer. In this case, on the first sub-model, extracting target class text features corresponding to the first text classification task from the text feature information includes:

[0211] The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text features corresponding to the first text classification task.

[0212] In an embodiment of the present disclosure, the first sub-model may include multiple levels of first fully connected layers, and an activation function is arranged between two adjacent levels of the first fully connected layers.

[0213] In the disclosed embodiment, the first sub-model further includes a cascaded second fully connected layer and a normalized index. In this case, predicting a first text classification result corresponding to the first text classification task according to the target class text feature includes:

[0214] The target class text features are processed on the second fully connected layer and the normalized index to obtain the first text classification result corresponding to the first text classification task.

[0215] In the disclosed embodiment, the second sub-model includes a cascaded third fully connected layer and an activation function. In this case, predicting the second text classification result corresponding to the second text classification task according to the concatenated text features includes:

[0216] The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification task.

[0217] In an embodiment of the present disclosure, when the target text data includes a digital sequence, predicting a first text classification result corresponding to the first text classification task according to the target class text feature includes:

[0218] A first target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification task is predicted for the first target feature vector on the first sub-model.

[0219] After passing through the encoder and the first sub-model, the target feature vector corresponding to the first digit sequence in the target class text feature actually contains the overall context information, that is, the text information of all subsequent digit sequences. Therefore, the target feature vector corresponding to the first digit sequence can be used to predict the first text classification result.

[0220] In an embodiment of the present disclosure, when the target text data includes a digital sequence, predicting a second text classification result corresponding to the second text classification task according to the concatenated text feature includes:

[0221] A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second text classification is predicted for the second target feature vector on the second sub-model.

[0222] Figure 6 Flow chart of the text classification method proposed in the embodiment of the present disclosure, such as Figure 6 As shown, the text classification method may include the following steps:

[0223] Step 610: Encode the target text data on the encoder to obtain text feature information;

[0224] Step 620: extracting target text features corresponding to the first text classification task from the text feature information on the first sub-model, and predicting a first text classification result corresponding to the first text classification task based on the target text features;

[0225] Step 630: On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and a second text classification result corresponding to the second text classification task is predicted based on the concatenated text feature;

[0226] Step 640: When the target text data includes at least two sub-text data, and the second text classification label sets contained in the second text classification results of two adjacent sub-texts have an intersection, the two adjacent sub-texts are merged, and the union of the second text classification label sets of the two adjacent sub-texts is taken, and the union is used as the second text classification result of the merged sub-text.

[0227] In this case, the merging operation stops until the adjacent subtexts cannot be merged.

[0228] Figure 7 This is a structural diagram of a text classification system proposed in an embodiment of the present disclosure, wherein the text classification system includes the following four parts:

[0229] Data preprocessing;

[0230] Data augmentation;

[0231] Model training;

[0232] Model reasoning.

[0233] The following takes a comment text of any length as the corresponding text as an example to explain the principle of the text classification system. Among them, the comment text is labeled with the label of the event to which it belongs and the sentiment label. The first text classification task is the sentiment classification task, and the second text classification task is the event cause classification task.

[0234] (1) Data preprocessing of the comment text is divided into five steps, as follows:

[0235] The first step is to convert the unicode codes of the Emoji in the review text into their corresponding Chinese interpretation tables through a mapping table, so that the text classification model can parse the meaning of the Emoji expressions.

[0236] The second step is to find the positive and negative floating point numbers in the text through the regular expression "(-?\d+)(\.\d+)" and replace them with the character "f";

[0237] The third step is to replace the list symbol. The regular expression "\d+\." is used to replace the list number "1." with the symbol "." representing the separator.

[0238] The fourth step is to convert English punctuation into Chinese punctuation;

[0239] The fifth step is to replace the symbol representing the end of the sentence with "." to facilitate subsequent text segmentation.

[0240] (2) The second part is data enhancement. Based on the pre-set maximum text length max_seq, the pre-processed comment text is segmented according to ".". If the segmented text has no content or only punctuation, it can be directly discarded.

[0241] Determine whether the length of the segmented subtext exceeds max_sqe. If the subtext length exceeds max_sqe, use "," to split the subtext into smaller subtexts. All the split subtexts are assigned the event cause label and sentiment label corresponding to the atomic text, and made into independent samples, and the samples are split into training set, validation set and test set.

[0242] (3) The third part is text classification model training, which specifically includes the following steps.

[0243] 1) Use the tokenizer to process the training set, obtain the corresponding numeric sequence input_ids, segment sequence segment_ids and mask sequence mask_ids, and make batch data with batch size b.

[0244] 2) Input the batch data consisting of the digital sequence, the fragment sequence and the mask sequence into the pre-trained model based on the transformer-based bidirectional encoder representation technology (BERT) to obtain the hidden layer output H1∈R of the last layer b×max_seq×hidden , where max_sqe is the maximum sequence length and hidden is the hidden layer length.

[0245] 3) Input H1 into the Self-Attention layer and obtain:

[0246]

[0247] Where Q = W Q H1, K = W K H1, V = W V H1, d=hidden, is the hidden layer length.

[0248] Then input to the output layer consisting of three fully connected layers:

[0249] A passes through the first fully connected layer and obtains O1=Dense1(A)∈R b×max_seq×hidden , and use the sigmoid function to activate O1;

[0250] After activation by the sigmoid function, the second fully connected layer is input to obtain O2 = Dense2(O1)∈R b ×max_seq×2hidden , and activated using the linear rectification ReLU function;

[0251] After activation by the ReLU function, the third fully connected layer is input and activated with Sigmoid to obtain H2 = Sigmoid(Dense3(O2))∈R b×max_seq×hidden , the formula is as follows:

[0252] H2=Sigmoid(Dense3(ReLU(Dense2(Sigmoid(Dense1(A))))))

[0253] b is the batch size, and max_seq is the maximum sequence length of the text.

[0254] 4) Through H2 cls =H2[:,0], obtain the matrix formed by the vectors corresponding to all first digit sequences, pass through the sentiment classification fully connected layer, and perform softmax normalization to obtain the sentiment probability matrix:

[0255] P emo ∈R b×Nemo , where Nemo is the number of emotion types. emo Perform logarithmic cross entropy with the hot encoding of the true emotional label to obtain the emotional loss function L emo .

[0256] Among them, the emotion probability matrix is ​​composed of the probabilities corresponding to multiple emotion types.

[0257] 5) H2 cls The matrix H1 formed by the vectors corresponding to all the first digit sequences in the output H1 of the pre-trained model cls , concatenated into a matrix H cls ∈R b×2hidden , the concatenation matrix H cls To correspond to the above concatenated text features;

[0258] The concatenation matrix H cls Input to the fully connected layer for event type classification;

[0259] Use the sigmoid activation function to activate and obtain the probability matrix P of the event type label e ;

[0260] For the probability matrix P e Calculate the cross entropy loss L for event cause classification e .

[0261] 6) L emo With L e Merged into L through a trainable parameter w,

[0262] L=pw L emo +(1-p w )L e

[0263] where p w =sigmoid(w)∈(0,1)

[0264] 7) Refer to the dotted arrow and propagate the loss of L in reverse to update the trainable parameters of the model. After a certain period, fix the value of w and fine-tune the model until the effect of the model on the validation set does not get better, and stop training;

[0265] 8) Set a classification threshold vector T for the event type e ∈R Ne And initialized to 0.5, inference was performed on the test set, and T was set using grid search. e The thresholds of each label in the test set are gradually increased with a minimum step size of 0.01, and the threshold vector T that maximizes the F1 value of the test set data is selected. o ∈R Ne .

[0266] (4) The fourth part is the model reasoning part, please refer to Figure 8 ,The text classification method based on text classification model includes the following steps.

[0267] 1) The input is an unprocessed review text of any length. After data preprocessing and data sentence segmentation operations, sub-texts are obtained.

[0268] 2) The subtext is converted into batch data through the word segmenter. The model inputs the batch data into the trained text classification model to obtain the probability matrix P of the sentiment category. emo , and the probability matrix P of the event cause label e , compare Pe with the optimal threshold vector T obtained in 8) above o , if the threshold of a certain tag is exceeded, the subtext is determined to belong to this tag, and all tag IDs exceeding the threshold are recorded;

[0269] Emotion tag id emo The calculation formula is id = argmax (P emo ), record id emo The corresponding probability p emo .

[0270] 3) If the event type label sets corresponding to adjacent sub-texts contain intersections, the two sub-texts are merged, and the union of the two label sets is taken as the event type label set of the merged sub-text. The sentiment label with the highest probability in the sentiment classification result is recorded as the sentiment label of the merged text, until the adjacent sub-texts cannot be merged, and the merging operation is stopped.

[0271] 4) Output the sentences, event type label sets and sentiment labels whose merged event type label sets are not empty.

[0272] In the disclosed embodiment, the analysis of the comment text includes the classification of the cause of the event and the classification of the customer's emotions, and the type of the cause of the event itself has an emotional tendency, such as the label "the goods are not as advertised" tends to be negative. Therefore, the result of the emotional classification helps to improve the result of the event cause classification task. Jointly training related tasks can share knowledge between tasks, enable the model to learn the characteristics of sample generalization, reduce overfitting, and help migrate the text classification model to new tasks.

[0273] Therefore, the disclosed embodiment can quickly segment and extract content from comment texts of any length, and can simultaneously perform event cause classification and sentiment classification.

[0274] Using the disclosed embodiment, the micro average precision of event label classification can reach 87%, the micro average recall rate can reach 77%, and the micro average precision and recall rate of sentiment classification can both reach 90%. And under the same conditions, the deep learning architecture of multi-task training has better classification effect than the deep models of multiple individual tasks. The micro average precision of event label classification is improved by 5% compared with the deep model of single task, and the micro average recall rate is improved by 4% compared with the deep model of single task. The sentiment classification result is on par with the effect of the single task deep model.

[0275] Fig. 9 Schematic diagram of a module of an embodiment of the text classification model training system of the present invention. Fig. 9 As shown, the text classification model training system includes but is not limited to:

[0276] A first encoding module 910 encodes the text sample data on an encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different text classification tasks respectively;

[0277] The first text classification module 920 extracts target class text features corresponding to the first text classification label from the text feature information on the first sub-model, and predicts a first text classification result corresponding to the first text classification label based on the target class text features;

[0278] The second text classification module 930 concatenates the text feature information and the target class text feature on the second sub-model to obtain a concatenated text feature, and predicts a second text classification result corresponding to the second text classification label according to the concatenated text feature;

[0279] The training module 940 obtains the target loss function of the text classification model according to the first loss function between the first text classification result and the first text classification label, and the second loss function between the second text classification result and the second text classification label, and trains the text classification model using the target loss function until the training stop condition is reached.

[0280] In the disclosed embodiment, the model training system performs joint training for different types of text classification tasks. Among them, by splicing text feature information with target class text features, the target class text features learned by the first sub-model are shared with the second sub-model, and the second sub-model can learn more generalized features, reduce overfitting, and improve the accuracy of model results. At the same time, the training results of each sub-model will be reflected in the model parameters of each sub-model, thereby improving the accuracy of the model results of a single sub-model.

[0281] The implementation principle of the above modules can be found in the relevant introduction of the text classification model training method, which will not be repeated here.

[0282] In an alternative embodiment, the encoder is a transformer-based bidirectional encoder;

[0283] The first encoding module 910 is specifically used for:

[0284] The text sample data is input into the transformer-based bidirectional encoder to obtain text feature information.

[0285] In an optional embodiment, the first sub-model includes a cascaded self-attention layer and at least one first fully connected layer;

[0286] The first classification module 920 is specifically used for:

[0287] The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text feature corresponding to the first text classification label.

[0288] In an optional embodiment, the first sub-model further includes a cascaded second fully connected layer and a normalized index;

[0289] The first classification module 920 is specifically used for:

[0290] The target class text features are processed on the second fully connected layer and the normalized index to obtain a first text classification result corresponding to the first text classification label.

[0291] In an optional embodiment, the second sub-model includes a cascaded third fully connected layer and an activation function;

[0292] The second classification module 930 is specifically used for:

[0293] The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification label.

[0294] In an optional embodiment, when the text sample data includes a digital sequence, the first classification module 920 is specifically used to:

[0295] A target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification label is predicted for the target feature vector on the first sub-model.

[0296] In an optional embodiment, when the text sample data includes a digital sequence, the second classification module 930 is specifically used to:

[0297] A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second text classification label is predicted for the second target feature vector on the second sub-model.

[0298] In an optional embodiment, the text classification model further includes a word segmenter;

[0299] and Fig. 9 compared to, Fig.10 The text classification model training system also includes:

[0300] The first word segmentation module 1010 obtains a target text sample, performs word segmentation and sentence segmentation operations on the target text sample on a word segmenter, obtains a digital sequence corresponding to the word segmentation operation, obtains a fragmentation sequence including the digital sequence corresponding to the sentence segmentation operation, and obtains text sample data according to the fragmentation sequence.

[0301] In an optional embodiment, the first word segmentation module 1010 is specifically used for:

[0302] Obtaining an original text sample, segmenting the original text sample based on a sentence end identifier to obtain a sub-text sample;

[0303] When the text length of the sub-text sample is not shorter than the preset target text length, the sub-text sample is segmented to obtain a target text sample whose text length is shorter than the target text length.

[0304] In an alternative embodiment, Fig.10 compared to, Fig.11 The text classification model training system also includes:

[0305] The first data preprocessing module 1110 performs data preprocessing on the original text sample before segmenting the original text sample based on the sentence end identifier on the word segmenter to obtain the sub-text sample.

[0306] Fig.12 FIG. 1 is a module diagram of an embodiment of a text classification system based on a text classification model of the present invention. Fig.12 As shown, the text classification system based on the text classification model includes but is not limited to:

[0307] The second encoding module 1210 encodes the target text data on an encoder to obtain text feature information;

[0308] The first text classification module 1220 extracts target class text features corresponding to the first text classification task from the text feature information on the first sub-model, and predicts a first text classification result corresponding to the first text classification task based on the target class text features;

[0309] The second text classification module 1230 concatenates the text feature information and the target class text feature on the second sub-model to obtain a concatenated text feature, and predicts a second text classification result corresponding to the second text classification task based on the concatenated text feature.

[0310] In the disclosed embodiment, the text classification system shares the target class text features extracted by the first sub-model with the second sub-model by splicing the text feature information with the target class text features, and the second sub-model can use more generalized text features to predict the second text classification result, and the second text classification result has high accuracy. In addition, since the first sub-model and the second sub-model are cascaded together through the encoder, the text feature information extracted by the encoder has good feature generalization for both the first sub-model and the second sub-model, which to a certain extent makes the first text classification result and the second text classification result have good accuracy.

[0311] The implementation principle of the above modules can be found in the relevant introduction of text classification method, which will not be repeated here.

[0312] In an alternative embodiment, the encoder is a transformer-based bidirectional encoder;

[0313] The second encoding module 1210 is specifically used for:

[0314] The target text data is input into the transformer-based bidirectional encoder to obtain text feature information.

[0315] In an optional embodiment, the first sub-model includes a cascaded self-attention layer and at least one first fully connected layer;

[0316] The first text classification module 1220 is specifically used for:

[0317] The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text features corresponding to the first text classification task.

[0318] In an optional embodiment, the first sub-model further includes a cascaded second fully connected layer and a normalized index;

[0319] The first text classification module 1220 is specifically used for:

[0320] The target class text features are processed on the second fully connected layer and the normalized index to obtain the first text classification result corresponding to the first text classification task.

[0321] In an optional embodiment, the second sub-model includes a cascaded third fully connected layer and an activation function;

[0322] The second text classification module 1230 is specifically used for:

[0323] The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification task.

[0324] In an optional embodiment, when the target text data includes a digital sequence, the first text classification module 1220 is specifically configured to:

[0325] A first target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification task is predicted for the first target feature vector on the first sub-model.

[0326] In an optional embodiment, when the target text data includes a digital sequence, the second text classification module 1230 is specifically used to:

[0327] A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second text classification is predicted for the second target feature vector on the second sub-model.

[0328] In an optional embodiment, the text classification model further includes a word segmenter;

[0329] and Fig.12 compared to, Fig.13 The text classification system shown also includes:

[0330] The second word segmentation module 1310 obtains a text to be processed before encoding the target text data on the encoder, performs word segmentation and sentence segmentation operations on the text to be processed on the word segmenter, obtains a digital sequence corresponding to the word segmentation operation, obtains a fragmentation sequence including the digital sequence corresponding to the sentence segmentation operation, and obtains the target text data according to the fragmentation sequence.

[0331] In an optional embodiment, the second word segmentation module 1310 is specifically used for:

[0332] Obtaining an original text, segmenting the original text based on sentence end identifiers to obtain sub-texts;

[0333] When the text length of the subtext is not shorter than the preset target text length, the subtext is segmented to obtain a to-be-processed text whose text length is shorter than the target text length.

[0334] In an alternative embodiment, Fig.13 compared to, Fig.14 The text classification system shown also includes:

[0335] The second data preprocessing module 1410 performs data preprocessing on the original text before segmenting the original text based on the sentence end identifier to obtain subtexts.

[0336] In an alternative embodiment, Fig.12 compared to, Fig.15 The text classification system shown also includes:

[0337] The text merging module 1510 merges the two adjacent subtexts when the target text data includes at least two subtext data and the second text classification label sets contained in the second text classification results of the two adjacent subtexts have an intersection, and takes the union of the second text classification label sets of the two adjacent subtexts, and uses the union as the second text classification result of the merged subtexts.

[0338] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein executable instructions of the processor are stored, wherein the processor is configured to execute the text classification model training method or the steps of the text classification method by executing the executable instructions.

[0339] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "text processing platforms".

[0340] Fig.16 Schematic diagram of the structure of the electronic device of the present invention. Fig.16 16A and 16B are used to describe the electronic device 1600 according to this embodiment of the present invention. Fig.16 The electronic device 1600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0341] like Fig.16As shown, the electronic device 1600 is in the form of a general computing device. The components of the electronic device 1600 may include, but are not limited to: at least one processing unit 1610, at least one storage unit 1620, a bus 1630 connecting different text processing platform components (including the storage unit 1620 and the processing unit 1610), a display unit 1640, etc.

[0342] The storage unit stores program codes, which can be executed by the processing unit 1610, so that the processing unit 1610 performs the steps of various exemplary embodiments of the present invention described in the text classification model training method or the text classification method section of this specification. For example, the processing unit 1610 can perform the following steps: Figure 1-Figure 6 Follow the steps shown in .

[0343] The storage unit 1620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1621 and / or a cache memory unit 1622 , and may further include a read-only memory unit (ROM) 1623 .

[0344] The storage unit 1620 may also include a program / utility 1624 having a set (at least one) of program modules 1625, such program modules 1625 including but not limited to: a processing system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0345] Bus 1630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0346] The electronic device 1600 may also communicate with one or more external devices 1670 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 1600, and / or any device that enables the electronic device 1600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1650.

[0347] Furthermore, the electronic device 1600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1660. The network adapter 1660 may communicate with other modules of the electronic device 1600 via the bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage text processing platforms, etc.

[0348] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the text classification model training method or the text classification method implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above text classification model training method or text classification method section of this specification.

[0349] The program product for implementing the above method according to an embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0350] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0351] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0352] Program code for performing the processes of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0353] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A text classification model training method, characterized in that: The text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder; The text classification model training method comprises: Encoding the text sample data on the encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different types of text classification tasks respectively; On the first sub-model, extracting target class text features corresponding to the first text classification label from the text feature information, and predicting a first text classification result corresponding to the first text classification label based on the target class text features; On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and a second text classification result corresponding to the second text classification label is predicted according to the concatenated text feature; A target loss function of the text classification model is obtained according to a first loss function between the first text classification result and the first text classification label, and a second loss function between the second text classification result and the second text classification label, and the text classification model is trained using the target loss function until a training stop condition is reached.

2. The text classification model training method according to claim 1, characterized in that: The text classification model also includes a word segmenter; Before encoding the text sample data on the encoder, the text classification model training method further includes: A target text sample is obtained, and word segmentation and sentence segmentation operations are performed on the target text sample on the word segmenter, a digital sequence is obtained corresponding to the word segmentation operation, a fragmentation sequence including the digital sequence is obtained corresponding to the sentence segmentation operation, and the text sample data is obtained according to the fragmentation sequence.

3. The text classification model training method according to claim 2, characterized in that: The obtaining of the target text sample comprises: Obtaining an original text sample, and segmenting the original text sample based on a sentence end identifier to obtain a sub-text sample; In the case that the text length of the sub-text sample is not shorter than the preset target text length, the sub-text sample is segmented to obtain the target text sample whose text length is shorter than the target text length.

4. The text classification model training method according to claim 3, characterized in that: Before segmenting the original text sample based on the sentence end identifier to obtain the sub-text sample, the text classification model training method further includes: Data preprocessing is performed on the original text sample.

5. The text classification model training method according to claim 1, characterized in that: The encoder is a transformer-based bidirectional encoder; The text sample data is encoded on the encoder to obtain text feature information, including: The text sample data is input into a transformer-based bidirectional encoder to obtain the text feature information.

6. The text classification model training method according to claim 1, characterized in that: The first sub-model includes a cascaded self-attention layer and at least one first fully connected layer; Extracting target class text features corresponding to the first text classification label from the text feature information on the first sub-model includes: The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain the target class text feature corresponding to the first text classification label.

7. The text classification model training method according to claim 6, characterized in that: The first sub-model also includes a cascaded second fully connected layer and a normalized index; Predicting a first text classification result corresponding to the first text classification label according to the target class text feature includes: The target class text feature is processed on the second fully connected layer and the normalized index to obtain a first text classification result corresponding to the first text classification label.

8. The text classification model training method according to claim 1, characterized in that: The second sub-model includes a cascaded third fully connected layer and an activation function; Predicting a second text classification result corresponding to the second text classification label according to the concatenated text feature includes: The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification label.

9. The text classification model training method according to claim 1, characterized in that: In a case where the text sample data includes a digital sequence, predicting a first text classification result corresponding to the first text classification label according to the target class text feature includes: A target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification label is predicted for the target feature vector on the first sub-model.

10. The text classification model training method according to claim 1, characterized in that: In a case where the text sample data includes a digital sequence, predicting a second text classification result corresponding to the second text classification label based on the concatenated text features on the second sub-model includes: A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text feature, and a second text classification result corresponding to the second text classification label is predicted for the second target feature vector on the second sub-model.

11. A text classification method based on a text classification model, characterized in that: The text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder; The text classification method comprises: Encoding the target text data on the encoder to obtain text feature information; On the first sub-model, extracting target class text features corresponding to a first text classification task from the text feature information, and predicting a first text classification result corresponding to the first text classification task based on the target class text features; On the second sub-model, the text feature information and the target class text feature are concatenated to obtain a concatenated text feature, and a second text classification result corresponding to the second text classification task is predicted based on the concatenated text feature.

12. The text classification method based on the text classification model according to claim 11, characterized in that: The text classification model also includes a word segmenter; The text classification method also includes: Before encoding the target text data on the encoder, a text to be processed is obtained, and word segmentation and sentence segmentation operations are performed on the text to be processed on the word segmenter, a digital sequence is obtained corresponding to the word segmentation operation, and a fragmentation sequence including the digital sequence is obtained corresponding to the sentence segmentation operation, and the target text data is obtained according to the fragmentation sequence.

13. The text classification method based on the text classification model according to claim 12, characterized in that: The obtaining of the text to be processed comprises: Obtaining an original text, and segmenting the original text based on sentence end identifiers to obtain subtexts; In the case that the text length of the sub-text is not shorter than the preset target text length, the sub-text is segmented to obtain the to-be-processed text whose text length is shorter than the target text length.

14. The text classification method based on the text classification model according to claim 13, characterized in that: Before segmenting the original text based on the sentence end identifier to obtain subtexts, the text classification method further includes: Data preprocessing is performed on the original text.

15. The text classification method based on the text classification model according to claim 11, characterized in that: The encoder is a transformer-based bidirectional encoder; The target text data is encoded on the encoder to obtain text feature information, including: The target text data is input into the transformer-based bidirectional encoder to obtain the text feature information.

16. The text classification method based on the text classification model according to claim 11, characterized in that: The first sub-model includes a cascaded self-attention layer and at least one first fully connected layer; On the first sub-model, extracting target class text features corresponding to the first text classification task from the text feature information includes: The text feature information is processed sequentially on the self-attention layer and at least one first fully connected layer to obtain target class text features corresponding to the first text classification task.

17. The text classification method based on the text classification model according to claim 11, characterized in that: The first sub-model also includes a cascaded second fully connected layer and a normalized index; Predicting a first text classification result corresponding to the first text classification task according to the target class text feature includes: The target class text features are processed on the second fully connected layer and the normalized index to obtain a first text classification result corresponding to the first text classification task.

18. The text classification method based on the text classification model according to claim 11, characterized in that: The second sub-model includes a cascaded third fully connected layer and an activation function; Predicting a second text classification result corresponding to the second text classification task according to the concatenated text features includes: The concatenated text features are processed on the third fully connected layer and the activation function to obtain a second text classification result corresponding to the second text classification task.

19. The text classification method based on the text classification model according to claim 11, characterized in that: In a case where the target text data includes a digital sequence, predicting a first text classification result corresponding to the first text classification task according to the target class text feature includes: A first target feature vector corresponding to the first digit sequence is obtained from the target class text feature, and a first text classification result corresponding to the first text classification task is predicted for the first target feature vector on the first sub-model.

20. The text classification method based on the text classification model according to claim 11, characterized in that: In the case where the target text data includes a digital sequence, predicting a second text classification result corresponding to the second text classification task according to the concatenated text feature includes: A second target feature vector corresponding to the first digit sequence is obtained from the concatenated text features, and a second text classification result corresponding to the second type of text classification is predicted for the second target feature vector on the second sub-model.

21. The text classification method based on the text classification model according to claim 11, characterized in that: The text classification method also includes: If the target text data includes at least two sub-text data, and the second text classification label sets contained in the second text classification results of two adjacent sub-texts have an intersection, the two adjacent sub-texts are merged, and the union of the second text classification label sets of the two adjacent sub-texts is taken, and the union is used as the second text classification result of the merged sub-text.

22. A text classification model training system, characterized in that: The text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder; The text classification model training system comprises: A first encoding module, encoding the text sample data on the encoder to obtain text feature information, wherein the text sample data carries a first text classification label and a second text classification label, and the first text classification label and the second text classification label correspond to different text classification tasks respectively; A first text classification module, which extracts target class text features corresponding to the first text classification label from the text feature information on the first sub-model, and predicts a first text classification result corresponding to the first text classification label based on the target class text features; A second text classification module, on the second sub-model, concatenates the text feature information and the target class text feature to obtain a concatenated text feature, and predicts a second text classification result corresponding to the second text classification label according to the concatenated text feature; A training module obtains a target loss function of the text classification model according to a first loss function between the first text classification result and the first text classification label, and a second loss function between the second text classification result and the second text classification label, and trains the text classification model using the target loss function until a training stop condition is reached.

23. A text classification system based on a text classification model, characterized in that: The text classification model includes an encoder and a first sub-model and a second sub-model that share the encoder; The text classification system comprises: A second encoding module encodes the target text data on the encoder to obtain text feature information; A first text classification module, which extracts target class text features corresponding to a first text classification task from the text feature information on the first sub-model, and predicts a first text classification result corresponding to the first text classification task based on the target class text features; The second text classification module concatenates the text feature information and the target class text feature on the second sub-model to obtain a concatenated text feature, and predicts a second text classification result corresponding to the second text classification task based on the concatenated text feature.

24. An electronic device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to execute the steps of the text classification model training method described in any one of claims 1 to 10, or the text classification method based on the text classification model described in any one of claims 11 to 21 by executing the executable instructions.

25. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the text classification model training method described in any one of claims 1 to 10, or the text classification method based on the text classification model described in any one of claims 11 to 21 are implemented.

Images