Text multi-label classification method and device

Abstract

The present specification relates to the technical field of natural language processing and artificial intelligence, and specifically discloses a text multi-label classification method and device, wherein the method comprises: receiving a text multi-classification request; the target text data is carried in the multi-classification request; inputting the target text data into a pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data; generating a prompt template matrix based on a pre-constructed adaptive prompt template; the adaptive prompt template is constructed through iterative training based on contrastive learning; splicing the prompt template matrix and the target pre-training word vector matrix to obtain a spliced target sentence vector representation; inputting the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data; and mapping and classifying the target sentence representation using a linear classifier to obtain a label set corresponding to the target text data. The above method can improve the accuracy and efficiency of multi-label classification.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of natural language processing and artificial intelligence, and particularly relates to a text multi-label classification method and device. BACKGROUND

[0002] The text multi-label classification task refers to that there are multiple correct labels for a sentence. Multi-label classification tasks exist in a large number of real applications. For example, Wikipedia has more than 1 million category labels, and an article has more than one related label. The webpage of “potato” has labels such as “solanum tuberosum”, “root vegetable”, and “crop from South America”.

[0003] However, in the multi-label classification task, the label space is large, and there is a serious data sparsity problem. The data set usually shows the characteristics of long-tail distribution, which means that a considerable part of the data label has very little related training data. Without additional manual labeling and data accumulation costs, there are problems such as poor data quality, less labeled data, and unbalanced label distribution. This data sparsity limits the accuracy of multi-label classification.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] Embodiments of the present specification provide a text multi-label classification method and device to solve the problem of low accuracy of multi-label classification in the prior art.

[0006] Embodiments of the present specification provide a text multi-label classification method, which comprises:

[0007] receiving a text multi-classification request; the multi-classification request carries target text data;

[0008] inputting the target text data into a pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data; generating a prompt template matrix based on a pre-constructed adaptive prompt template; the adaptive prompt template is constructed by iterative training based on contrastive learning;

[0009] concatenating the prompt template matrix and the target pre-training word vector matrix to obtain a concatenated target sentence vector representation; inputting the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data;

[0010] mapping and classifying the target sentence representation by using a linear classifier to obtain a label set corresponding to the target text data.

[0011] In one embodiment, before receiving the multi-classification request, further comprising:

[0012] randomly initializing the prompt template to generate an initialization prompt template matrix;

[0013] obtaining a labeled data set; the labeled data set includes a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences;

[0014] constructing a first labeled set and a second labeled set based on the labeled data set; the label corresponding to the sentence in the first labeled set includes a first label; the label corresponding to the sentence in the second labeled set does not include the first label;

[0015] performing contrastive learning based on the first labeled set, the second labeled set and the initialization prompt template matrix, fine-tuning the parameters of the initialization prompt template to obtain an adaptive prompt template.

[0016] In one embodiment, the labeled data set is obtained, including:

[0017] obtaining an original labeled data set; the original labeled data set includes a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences;

[0018] performing data augmentation on the original labeled data set to obtain an augmented labeled data set.

[0019] In one embodiment, the contrastive learning based on the first labeled set, the second labeled set and the initialization prompt template matrix, fine-tuning the parameters of the initialization prompt template to obtain an adaptive prompt template, includes:

[0020] inputting the sentences in the first labeled set into the pre-training model to obtain a first pre-training word vector matrix set; inputting the sentences in the second labeled set into the pre-training model to obtain a second pre-training word vector matrix set;

[0021] concatenating the initialization prompt template with the first pre-training word vector matrix in the first pre-training word vector matrix set to obtain a first sentence vector representation set; concatenating the initialization prompt template with the second pre-training word vector matrix in the second pre-training word vector matrix set to obtain a second sentence word vector representation set;

[0022] inputting the first sentence vector representation in the first sentence vector representation set into the encoding model to obtain a first output set; inputting the second sentence vector representation in the second sentence vector representation set into the encoding model to obtain a second output set;

[0023] construct a contrastive learning loss function based on the first output set and the second output set; and perform multi-round iterative training on the initial prompt template based on the contrastive learning loss function by using a gradient descent algorithm to obtain an adaptive prompt template.

[0024] In one embodiment, the multi-round iterative training on the initial prompt template based on the contrastive learning loss function by using a gradient descent algorithm to obtain an adaptive prompt template comprises:

[0025] constructing a classifier loss function corresponding to the linear classifier;

[0026] constructing an overall loss function based on the contrastive learning loss function and the classifier loss function;

[0027] performing joint training on the classifier and the initial prompt template based on the overall loss function by using a gradient descent algorithm to obtain a trained linear classifier and an adaptive prompt template.

[0028] In one embodiment, the contrastive learning loss function is:

[0029]

[0030] wherein L cl is the contrastive learning loss function, K represents the number of sentences in the second annotation set, s(x, y) represents the cosine similarity of x and y, c is a first output in the first output set corresponding to a sentence in the first annotation set, c + is a first output in the first output set corresponding to another sentence in the first annotation set, is a second output in the second output set corresponding to the i-th sentence in the second annotation set, and τ is a constant adjustment factor.

[0031] In one embodiment, the overall loss function is:

[0032] L = aL BCE +bL cl

[0033] wherein L is the loss function, L BCE is a classifier loss function, L cl is a contrastive learning loss function, and a and b are constants.

[0034] The embodiments of the present specification also provide a text multi-label classification device, comprising:

[0035] a receiving module configured to receive a text multi-classification request, wherein the multi-classification request carries target text data;

[0036] The generating module is configured to input the target text data into a pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data; and generate a prompt template matrix based on a pre-constructed adaptive prompt template, wherein the adaptive prompt template is constructed based on contrast learning through iterative training;

[0037] The splicing module is configured to splice the prompt template matrix and the target pre-training word vector matrix to obtain a spliced target sentence vector representation; and input the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data.

[0038] The classification module is configured to map and classify the target sentence representation by using a linear classifier to obtain a label set corresponding to the target text data.

[0039] The embodiments of the present specification further provide a computer device, comprising a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the text multi-label classification method described in any of the above embodiments.

[0040] The embodiments of the present specification further provide a computer-readable storage medium having computer instructions stored thereon, wherein the instructions are executed to implement the steps of the text multi-label classification method described in any of the above embodiments.

[0041] In an embodiment of the present specification, a text multi-label classification method is provided, which can receive a text multi-classification request, the multi-classification request carrying target text data, input the target text data into a pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data, generate a prompt template matrix based on an adaptive prompt template constructed through iterative training based on contrastive learning, splice the prompt template matrix and the target pre-training word vector matrix to obtain a spliced target sentence vector representation, input the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data, and finally map and classify the target sentence representation using a linear classifier to obtain a label set corresponding to the target text data. In the above scheme, by using a pre-training model, excellent learning results can be achieved on tasks with data sparsity problems, and the learning tasks in many zero-sample and few-sample scenarios have been significantly improved. The prompt learning increases the prompt information on the basis of the pre-training model, and without significantly changing the model structure and parameters, the downstream task can be changed to a text generation task in the form of a cloze test, improving the effect of the pre-training model. The prompt learning can be applied to the special corpus of the banking industry, and the multi-label classification problem in the question system of the bank can be solved by using the prompt learning. Further, by using the contrastive learning method, the matching of the adaptive template and the task can be improved, the prompt learning template suitable for the task is generated for specific multi-label classification problems, the problem of the prompt learning template affecting the accuracy of the result is solved, and the final multi-label classification effect can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] The drawings described herein are used to provide further understanding of the present specification, form a part of the present specification, and do not constitute a limitation of the present specification. In the drawings:

[0043] Figure 1 A flowchart of a text multi-label classification method in an embodiment of the present specification is shown;

[0044] Figure 2 A flowchart of a text multi-label classification method in an embodiment of the present specification is shown;

[0045] Figure 3 A schematic diagram of a text multi-label classification device in an embodiment of the present specification is shown;

[0046] Figure 4 A schematic diagram of a computer device in an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0047] The principles and spirit of the present specification will be described below with reference to a number of exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present specification, and in no way limit the scope of the present specification. On the contrary, these embodiments are provided to make the present specification disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] Those skilled in the art know that the embodiments of the present specification can be implemented as a system, device, method or computer program product. Therefore, the present specification can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0049] The embodiment of the present specification provides a text multi-label classification method. The method in the present embodiment can be applied to a server. In this application scenario, the server can receive a text multi-classification request sent by a client. The text multi-classification request can carry target text data. The target text data can be a sentence to be classified. In one embodiment, the method in the present embodiment can be applied to an intelligent question and answer system. After receiving the target text data input by the user, the target text data can be classified.

[0050] In response to the text multi-classification request, the server can input the target text data into a pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data. The pre-training model here can include one of the following: a Bert model, a GPT model, a T5 model, and the like. The server can also generate a prompt template matrix based on a pre-constructed adaptive prompt template. The adaptive prompt template here can be constructed by iterative training based on contrastive learning.

[0051] The server can concatenate the target pre-training word vector matrix and the prompt template matrix to obtain a target sentence vector representation corresponding to the target text data. Then, the server can input the target sentence vector representation into an encoding model for encoding to obtain a target sentence representation corresponding to the target text data. The encoding model here can include various models such as a Transformer model and an LSTM model. The server can map and classify the target sentence representation using a linear classifier to obtain a label set corresponding to the target text data. The label set can include multiple labels to which the target text data belongs. The server can return the label set to the client.

[0052] In the above scenario example, by utilizing the pre-training model, excellent learning results can be achieved on tasks with data sparsity problems, and the learning tasks in many zero-shot and few-shot scenarios have significantly improved effects, prompting learning on the basis of the pre-training model to increase the prompt information, without significantly changing the model structure and parameters, the downstream task can be changed to a fill-in-the-blank text generation task, improving the effect of the pre-training model, and the prompt learning can be applied to the special corpus of the banking industry, and the multi-label classification problem in the bank question system can be solved by using the prompt learning method. Further, by using the contrast learning method, the matching of the adaptive template and the task can be improved, and for specific multi-label classification problems, a prompt learning template that is adapted to the task is generated, solving the problem of the influence of the prompt learning template on the accuracy of the results, and effectively improving the final multi-label classification effect.

[0053] Figure 1 A flowchart of a text multi-label classification method according to an embodiment of the present specification is shown. Although the present specification provides method operation steps or device structures as described in the following embodiments or shown in the accompanying drawings, more or fewer operation steps or module units can be included in the method or device based on conventional or non-inventive labor. In steps or structures that do not have essential causal relationships in logic, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiment description and the accompanying drawings of the present specification. When the method or module structure is applied in actual device or terminal product, it can be sequentially executed or executed in parallel (for example, parallel processor or multi-thread processing environment, even distributed processing environment) according to the method or module structure shown in the embodiment or the accompanying drawings.

[0054] Specifically, as shown in Figure 1 The text multi-label classification method provided by an embodiment of the present specification can include the following steps:

[0055] Step S101, receiving a text multi-classification request; the multi-classification request carries target text data.

[0056] The method in the embodiment can be applied to a server. The server can receive a text multi-classification request sent by a client. The text multi-classification request can include target text data. The target text data can include sentence data to be classified. In an embodiment, the method in the embodiment can be applied to an intelligent question and answer system. After receiving the target text data input by the user, the target text data can be classified.

[0057] Step S102, input the target text data into the pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data; generate a prompt template matrix based on a pre-constructed adaptive prompt template; the adaptive prompt template is constructed based on contrast learning through iterative training.

[0058] In response to a text multi-classification request, the server can input the target text data into the pre-training model for feature extraction to obtain a target pre-training word vector matrix corresponding to the target text data. In an embodiment, the pre-training model can be a natural language processing pre-training model, which can include one of the following: a Bert model, a GPT model, a T5 model, and the like.

[0059] The server can generate a prompt template matrix based on a pre-constructed adaptive prompt template. The adaptive prompt template here can be constructed based on contrast learning through iterative training.

[0060] Step S103, concatenate the prompt template matrix and the target pre-training word vector matrix to obtain a concatenated target sentence vector representation; input the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data.

[0061] The server can concatenate the prompt template matrix and the target pre-training word vector matrix to obtain a concatenated target sentence vector representation. Then, the server can input the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data. The encoding model can encode the target sentence vector representation. The encoding model can include various models such as a Transformer model and an LSTM model.

[0062] Step S104, map and classify the target sentence representation using a linear classifier to obtain a label set corresponding to the target text data.

[0063] After obtaining the target sentence representation, the linear classifier can be used to map and classify the target sentence representation. The linear classifier can be a multi-layer perceptron. The input dimension of the linear classifier is the same as the dimension of the target sentence representation, the output dimension is the number of categories, and the sigmoid function is used for activation, which can obtain the possibility of belonging to each label. By setting a threshold, a label set corresponding to the target text data can be generated. After obtaining the label set, the label set corresponding to the target text data can be returned to the client.

[0064] In the above embodiments, by using the pre-training model, excellent learning results can be achieved on tasks with data sparsity problems, and the learning tasks in many zero-sample and few-sample scenarios have been significantly improved, prompting learning based on the pre-training model to add prompt information, without significantly changing the model structure and parameters, the downstream task can be changed to a fill-in-the-blank text generation task, improving the effect of the pre-training model, and the prompt learning can be applied to the special corpus of banks and other industries, and the multi-label classification problem in the bank question system can be solved by prompt learning. Further, by using the contrast learning method, the matching of the adaptive template and the task can be improved, and the prompt learning template suitable for the task can be generated for specific multi-label classification problems, solving the problem of the influence of the prompt learning template on the accuracy of the results, and effectively improving the multi-label classification effect.

[0065] In some embodiments of the present specification, before receiving the multi-classification request, the method can further include: randomly initializing the prompt template to generate an initialized prompt template matrix; obtaining a labeled data set; the labeled data set includes a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences; constructing a first label set and a second label set based on the labeled data set; the labels corresponding to the sentences in the first label set include a first label; the labels corresponding to the sentences in the second label set do not include the first label; and performing contrast learning based on the first label set, the second label set, and the initialized prompt template matrix to fine-tune the parameters of the initialized prompt template to obtain an adaptive prompt template.

[0066] Specifically, the server can randomly initialize the prompt template to generate an initialized prompt template matrix. The server can obtain a labeled data set. The labeled data set can include a plurality of text data (or sentences) and at least one label corresponding to each sentence in the plurality of sentences. A first label set and a second label set can be constructed based on the labeled data set. The labels corresponding to the sentences in the first label set can all include a first label. The first label can be any one of the plurality of labels. The labels corresponding to the sentences in the second label set do not include the first label. Then, contrast learning can be performed based on the first label set, the second label set, and the initialized prompt template matrix to fine-tune the parameters of the initialized prompt template to obtain an adaptive prompt template. Contrast learning is a self-supervised learning method that can learn the general features of a data set by learning the similarities and differences of the data without labels. In this way, the matching of the adaptive template and the classification task can be improved, and a prompt learning template suitable for the task can be generated to further improve the accuracy of the classification.

[0067] In some embodiments of the present specification, obtaining the labeled data set can include: obtaining an original labeled data set; the original labeled data set includes a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences; and performing data augmentation on the original labeled data set to obtain an augmented labeled data set. Specifically, the server can select a label corresponding to a number of sentences less than a first preset number in the original labeled data set, perform data augmentation on the sentences corresponding to the selected label to generate new sentences, expand the standard data set, and ensure that the number of sentences corresponding to each label in the expanded labeled data set is not less than a second preset number. Through data augmentation, the problem of data sparsity can be effectively solved, and the accuracy of multi-label classification can be further improved.

[0068] In some embodiments of the present specification, based on the first label set, the second label set and the initialization prompt template matrix, the parameters of the initialization prompt template are fine-tuned through contrastive learning to obtain an adaptive prompt template, which can include: inputting the sentences in the first label set into the pre-training model to obtain a first pre-training word vector matrix set; inputting the sentences in the second label set into the pre-training model to obtain a second pre-training word vector matrix set; concatenating the initialization prompt template with the first pre-training word vector matrix in the first pre-training word vector matrix set to obtain a first sentence vector representation set; concatenating the initialization prompt template with the second pre-training word vector matrix in the second pre-training word vector matrix set to obtain a second sentence vector representation set; inputting the first sentence vector representation in the first sentence vector representation set into the encoding model to obtain a first output set; inputting the second sentence vector representation in the second sentence vector representation set into the encoding model to obtain a second output set; constructing a contrastive learning loss function based on the first output set and the second output set; and based on the contrastive learning loss function, the initialization prompt template is iteratively trained through a gradient descent algorithm for multiple rounds to obtain an adaptive prompt template.

[0069] Specifically, the server can input the sentences in the first annotation set and the second annotation set into the pre-training model respectively to obtain the corresponding first pre-training word vector matrix set and the second pre-training word vector matrix set. Then, the initialization prompt template can be concatenated with the first pre-training word vector matrix and the second pre-training word vector matrix in the first pre-training word vector matrix set and the second pre-training word vector matrix set respectively to obtain the first sentence vector representation set and the second sentence word vector representation set. Then, the sentence word vector representation sets in the first sentence vector representation set and the second sentence word vector representation set are input into the encoding model to obtain the corresponding first output set and the second output set. The first output set and the second output set correspond to the sentence vector representations corresponding to two different labels. By comparing the sentence vector representations corresponding to the two different labels, the initialization prompt template is trained by the gradient descent algorithm for multiple rounds to obtain an adaptive prompt template. In the above manner, the contrastive learning loss function can be constructed for contrastive learning to generate the adaptive prompt template.

[0070] In some embodiments of the present specification, based on the contrastive learning loss function, the initialization prompt template is trained by the gradient descent algorithm for multiple rounds to obtain an adaptive prompt template, which can include: constructing a classifier loss function corresponding to the linear classifier; based on the contrastive learning loss function and the classifier loss function, constructing an overall loss function; based on the overall loss function, the classifier and the initialization prompt template are jointly trained by the gradient descent algorithm to obtain a trained linear classifier and an adaptive prompt template. In the above embodiment, the multi-label classification task based on prompt learning and the prompt learning template adaptive learning task based on contrastive learning are jointly trained for multi-task learning, which can effectively improve the final multi-label classification effect.

[0071] In some embodiments of the present specification, the contrastive learning loss function is:

[0072]

[0073] wherein, L cl is the contrastive learning loss function, K represents the number of sentences in the second annotation set, s(x, y) represents the cosine similarity of x and y, c is the first output in the first output set corresponding to a sentence in the first annotation set, c + is the first output in the first output set corresponding to another sentence in the first annotation set, is the second output in the second output set corresponding to the i-th sentence in the second annotation set, and the constant τ is a regulation coefficient.

[0074] In some embodiments of the present specification, the overall loss function is:

[0075] L = aL BCE +bL cl

[0076] wherein L is the loss function, L BCE is a classifier loss function, L cl is a contrastive learning loss function, and a and b are constants. In one embodiment, the classifier loss function can be a binary cross-entropy loss.

[0077] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. For specific description, reference can be made to the description of the related processing embodiments described above, which will not be repeated here.

[0078] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or advantageous.

[0079] The above method will be described in conjunction with a specific embodiment, however, it is worth noting that this specific embodiment is only for better illustration of the present specification, and does not constitute an improper limitation of the present specification.

[0080] A text multi-label classification method is proposed in the present specific embodiment. In the present specific embodiment, prompt learning is used to increase prompt information on the basis of a pre-trained model, and without significantly changing the model structure and parameters, the downstream task is changed to a text generation task in the form of a fill-in-the-blank, thereby improving the effect of the pre-trained model. The present embodiment applies the prompt model to a multi-label classification task to predict the labels of the problem. In order to better improve the effect of the prompt template, a contrastive learning method is further used to enhance the adaptive selection of the template, and two parts of losses, i.e., a prompt learning multi-label classification and a contrastive learning enhanced prompt learning template adaptive learning, are constructed, and after joint training, the final multi-label classification effect is improved.

[0081] For the input question Input, two special start and end markers [CLS] and [SEP] are added at the beginning and end respectively ([CLS] is a mark of the beginning of a sentence, and [SEP] is a mark of the end of a sentence). The entire sentence can be input into the pre-training model Bert to obtain a pre-training word vector matrix W, a Prompt template matrix P is randomly initialized, P and W are spliced, input into the Transformer, and the hidden layer output at the [CLS] position is taken as the sentence vector representation, denoted as c, and the entire process is denoted as c = Prompt Bert (Input).

[0082] After obtaining the sentence representation c, c is mapped and classified by a linear classifier multilayer perception (MLP). The input dimension of the classifier is the same as c, the output dimension is the number of categories, and the sigmoid function is used for activation. The possibility of belonging to each label can be obtained, and the final predicted label set is determined by setting a threshold t. The loss function is as follows:

[0083] L BCE =BCE(Y,σ(cM))

[0084] Where Y is the correct label vector, M is the classifier parameter matrix, BCE is the binary cross entropy loss, and sigma is the sigmoid function.

[0085] Figure 2 A flowchart of generating an adaptive prompt template based on contrastive learning is shown. According to the label, the sentence is reorganized to construct a sentence set belonging to the same label. For the sentence input Input, another sentence under the same label is randomly selected, and the sentence is spliced into the prompt template and input into Prompt Bert to obtain the positive pair sentence vector output c + . Randomly select a sentence with different labels in the data set as a negative example. In order to distinguish the original sentence input Input and the negative example sentence input, Input is denoted as Positive Input, and the negative example sentence is denoted as Negative Input. After Prompt Bert, the negative example sentence is encoded as c - , as shown in Figure 2 .

[0086] Overall, the inputs Input, Positive Input and Negative Input are encoded by Prompt Bert to obtain c, c + ,c - . The contrastive learning loss function is defined as follows:

[0087]

[0088] Wherein, K represents the number of negative examples randomly sampled for a positive example, s(x, y) represents the cosine similarity of x, y, and the constant τ is a regulating coefficient.

[0089] The overall loss of the model is:

[0090] L = aL BCE +bL cl

[0091] Wherein, a and b are weight coefficients.

[0092] The model is initialized using an open-source pre-trained model Bert, and the contrastive learning loss function is optimized using the mini-batch gradient descent method. After multiple rounds of iterative training, the Prompt template vector is fine-tuned to obtain a task-related adaptive Prompt template. Through joint training, a trained Transformer model, a linear classifier, and an adaptive prompt template can be obtained.

[0093] In this embodiment, the prompt learning model is first applied to a bank-specific corpus to solve the multi-label classification problem in the question-asking system using prompt learning. By using contrastive learning, the matching of the adaptive template and the task is improved, and a more task-adaptive prompt learning template is generated for specific multi-label classification problems, solving the problem of the influence of the prompt learning template on the accuracy of the results. The multi-label classification task based on prompt learning and the prompt learning template adaptive learning task based on contrastive learning are jointly trained for multi-task learning, which can effectively improve the final multi-label classification effect.

[0094] Based on the same inventive concept, the embodiments of the present specification also provide a text multi-label classification device, as described in the following embodiments. Since the principle of solving problems by the text multi-label classification device is similar to that of the text multi-label classification method, the implementation of the text multi-label classification device can be referred to the implementation of the text multi-label classification method, and the repeated parts will not be described here. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is conceived. Figure 3 is a structural block diagram of the text multi-label classification device of the embodiments of the present specification, as Figure 3 shown, comprising: a receiving module 301, a generating module 302, a splicing module 303, and a classification module 304, which will be described below.

[0095] The receiving module 301 is configured to receive a text multi-classification request; the multi-classification request carries target text data.

[0096] The generation module 302 is configured to input the target text data into a pre-training model to obtain a target pre-training word vector matrix corresponding to the target text data; and generate a prompt template matrix based on a pre-constructed adaptive prompt template. The adaptive prompt template is constructed based on contrast learning through iterative training.

[0097] The splicing module 303 is configured to splice the prompt template matrix and the target pre-training word vector matrix to obtain a spliced target sentence vector representation; and input the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data.

[0098] The classification module 304 is configured to map and classify the target sentence representation by using a linear classifier to obtain a label set corresponding to the target text data.

[0099] In some embodiments of the present specification, the apparatus can further include a construction module, which can include:

[0100] A generation unit configured to randomly initialize a prompt template to generate an initialized prompt template matrix.

[0101] An acquisition unit configured to acquire a labeled data set. The labeled data set includes a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences.

[0102] A construction unit configured to construct a first label set and a second label set based on the labeled data set. The label corresponding to the sentence in the first label set includes a first label. The label corresponding to the sentence in the second label set does not include the first label.

[0103] A contrast learning unit configured to perform contrast learning based on the first label set, the second label set, and the initialized prompt template matrix, fine-tune parameters of the initialized prompt template, and obtain an adaptive prompt template.

[0104] In some embodiments of the present specification, the acquisition unit can be specifically configured to:

[0105] Acquire an original labeled data set. The original labeled data set includes a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences.

[0106] Perform data augmentation on the original labeled data set to obtain an augmented labeled data set.

[0107] In some embodiments of the present specification, the contrast learning unit can be specifically configured to:

[0108] inputting the sentences in the first annotation set into the pre-training model to obtain a first pre-training word vector matrix set; inputting the sentences in the second annotation set into the pre-training model to obtain a second pre-training word vector matrix set;

[0109] concatenating the initialization prompt template and the first pre-training word vector matrix in the first pre-training word vector matrix set to obtain a first sentence vector representation set; and concatenating the initialization prompt template and the second pre-training word vector matrix in the second pre-training word vector matrix set to obtain a second sentence vector representation set;

[0110] inputting the first sentence vector representation in the first sentence vector representation set into the encoding model to obtain a first output set; and inputting the second sentence vector representation in the second sentence vector representation set into the encoding model to obtain a second output set;

[0111] constructing a contrastive learning loss function based on the first output set and the second output set; and performing multi-round iterative training on the initialization prompt template based on the contrastive learning loss function by using a gradient descent algorithm to obtain an adaptive prompt template.

[0112] In some embodiments of the present specification, performing multi-round iterative training on the initialization prompt template based on the contrastive learning loss function by using a gradient descent algorithm to obtain an adaptive prompt template can include:

[0113] constructing a classifier loss function corresponding to the linear classifier;

[0114] constructing an overall loss function based on the contrastive learning loss function and the classifier loss function;

[0115] performing joint training on the classifier and the initialization prompt template based on the overall loss function by using a gradient descent algorithm to obtain a trained linear classifier and an adaptive prompt template.

[0116] In some embodiments of the present specification, the contrastive learning loss function can be:

[0117]

[0118] wherein, L cl is the contrastive learning loss function, K represents the number of sentences in the second annotation set, s(x, y) represents the cosine similarity of x and y, c is a first output in the first output set corresponding to a sentence in the first annotation set, and c + is a first output in the first output set corresponding to another sentence in the first annotation set, For the second output in the second output set corresponding to the i th sentence in the second label set, the constant τ is an adjustment coefficient.

[0119] In some embodiments of the present specification, the overall loss function can be:

[0120] L = aL BCE + bL cl

[0121] Wherein, L is the loss function, L BCE is a classifier loss function, L cl is a contrastive learning loss function, and a and b are constants.

[0122] From the above description, it can be seen that the embodiments of the present specification achieve the following technical effects: by using the pre-training model, excellent learning results can be obtained on tasks with data sparsity problems, and the learning tasks in many zero-sample and few-sample scenarios have been significantly improved. Prompt learning increases the prompt information on the basis of the pre-training model, and without significantly changing the model structure and parameters, the downstream task can be changed to a text generation task of the completion fill-in-the-blank type, improving the effect of the pre-training model. The prompt learning can be applied to the special corpus of the banking industry, and the multi-label classification problem in the bank question system can be solved by using the prompt learning method. Further, by using the contrastive learning method, the matching of the adaptive template and the task can be improved, and for specific multi-label classification problems, a prompt learning template that adapts to the task is generated, solving the problem of the influence of the prompt learning template on the accuracy of the results, and effectively improving the final multi-label classification effect.

[0123] The present specification also provides a computer device, which can specifically refer to Figure 4 The computer device based on the text multi-label classification method provided by the embodiments of the present specification can specifically include an input device 41, a processor 42, and a memory 43. The memory 43 is used to store processor executable instructions. The processor 42 executes the instructions to implement the steps of the text multi-label classification method described in any of the above embodiments.

[0124] In the embodiment, the input device can be specifically one of the main devices for information exchange between the user and the computer system. The input device can include a keyboard, a mouse, a camera, a scanner, a light pen, a handwriting input board, a voice input device, etc.; the input device is used to input raw data and programs for processing the data into the computer. The input device can also obtain data transmitted by other modules, units, devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of, for example, a microprocessor or a processor and a computer readable medium storing computer readable program code (for example, software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASIC), programmable logic controllers, and embedded microcontrollers, and the like. The memory can be specifically a memory device used to save information in modern information technology. The memory can include multiple levels, and in a digital system, as long as it can save binary data, it can be a memory; in an integrated circuit, a circuit without a physical form and with a storage function is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0125] In the embodiment, the functions and effects realized by the computer device can be explained in comparison with other embodiments, and will not be repeated here.

[0126] The embodiment of the present specification also provides a computer storage medium based on a text multi-label classification method, which stores computer program instructions. When the computer program instructions are executed, the steps of the text multi-label classification method described in any of the above embodiments are realized.

[0127] In the embodiment, the storage medium includes but is not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface set according to the standard of the communication protocol, used for network connection communication.

[0128] In the embodiment, the functions and effects realized by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments, and will not be repeated here.

[0129] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present description can be realized by a general computing device, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or each module or step can be manufactured into an individual integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the embodiments of the present description are not limited to any specific combination of hardware and software.

[0130] It is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent to those of skill in the art upon reading the above description. The scope of the description should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full range of equivalents to which such claims are entitled. In this regard, the description contemplates that the components and / or steps recited in the claims can be combined in a number of ways to yield desired structures and / or methodologies. Thus, the description is not limited to the specific arrangements or mixtures of components or steps described herein.

[0131] Although the present description has been described with reference to preferred embodiments, workers skilled in the art will recognize that changes can be made in form and detail without departing from the spirit and the scope of the description. Therefore, the description should not be limited to the preferred embodiments described herein, but should be given the full scope permitted by the appended claims, along with all equivalents of the claims.

Claims

1. A method of text multi-label classification, characterized by, The method comprises the following steps: receiving a text multi-classification request; the multi-classification request carries target text data; inputting the target text data into a pre-trained model to obtain a target pre-trained word vector matrix corresponding to the target text data; generating a prompt template matrix based on a pre-constructed adaptive prompt template; the adaptive prompt template is constructed through iterative training based on contrastive learning; concatenating the prompt template matrix and the target pre-trained word vector matrix to obtain a concatenated target sentence vector representation; inputting the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data; mapping and classifying the target sentence representation using a linear classifier to obtain a label set corresponding to the target text data.

2. The text multi-label classification method of claim 1, wherein, Before receiving the multi-classification request, the method further comprises the following steps: randomly initializing a prompt template to generate an initialized prompt template matrix; obtaining a labeled data set; the labeled data set comprises a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences; constructing a first label set and a second label set based on the labeled data set; the labels corresponding to the sentences in the first label set comprise a first label; the labels corresponding to the sentences in the second label set do not comprise the first label; performing contrastive learning based on the first label set, the second label set, and the initialized prompt template matrix to fine-tune the parameters of the initialized prompt template to obtain an adaptive prompt template.

3. The text multi-label classification method of claim 2, wherein, Obtaining a labeled data set comprises: obtaining an original labeled data set; the original labeled data set comprises a plurality of sentences and at least one label corresponding to each sentence in the plurality of sentences; performing data augmentation on the original labeled data set to obtain an augmented labeled data set.

4. The text multi-label classification method of claim 2, wherein, Performing contrastive learning based on the first label set, the second label set, and the initialized prompt template matrix to fine-tune the parameters of the initialized prompt template to obtain an adaptive prompt template comprises: inputting the sentences in the first label set into the pre-trained model to obtain a first pre-trained word vector matrix set; inputting the sentences in the second label set into the pre-trained model to obtain a second pre-trained word vector matrix set; concatenating the initialized prompt template with the first pre-trained word vector matrix in the first pre-trained word vector matrix set to obtain a first sentence vector representation set; concatenating the initialized prompt template with the second pre-trained word vector matrix in the second pre-trained word vector matrix set to obtain a second sentence word vector representation set; inputting the first sentence vector representation in the first sentence vector representation set into the encoding model to obtain a first output set; inputting the second sentence vector representation in the second sentence vector representation set into the encoding model to obtain a second output set; constructing a contrastive learning loss function based on the first output set and the second output set; based on the contrastive learning loss function, performing multi-round iterative training on the initialized prompt template using a gradient descent algorithm to obtain an adaptive prompt template.

5. The text multi-label classification method of claim 4, wherein, Based on the contrast learning loss function, a gradient descent algorithm is used to perform multi-round iterative training on the initialized prompt template to obtain an adaptive prompt template, including: Constructing a classifier loss function corresponding to the linear classifier; Based on the contrast learning loss function and the classifier loss function, an overall loss function is constructed; Based on the overall loss function, a gradient descent algorithm is used to jointly train the classifier and the initialized prompt template to obtain a trained linear classifier and an adaptive prompt template.

6. The text multi-label classification method of claim 4, wherein, The contrast learning loss function is: wherein L cl is the contrastive learning loss function, K represents the number of sentences in the second annotation set, s(x, y) represents the cosine similarity of x and y, c is the first output in the first output set corresponding to a sentence in the first annotation set, c + is the first output in the first output set corresponding to another sentence in the first annotation set, is the second output in the second output set corresponding to the i-th sentence in the second annotation set, and τ is a constant adjustment factor.

7. The text multi-label classification method of claim 5, wherein, The overall loss function is: L = aL + bL BCE L = aL + bL cl where L is the loss function, L BCE is a classifier loss function, L cl is a contrastive learning loss function, and a and b are constants.

8. A text multi-label classification apparatus characterized by comprising: Including: A receiving module for receiving a text multi-classification request; The multi-classification request carries target text data; A generation module for inputting the target text data into a pre-trained model to obtain a target pre-trained word vector matrix corresponding to the target text data; Based on a pre-constructed adaptive prompt template, a prompt template matrix is generated; The adaptive prompt template is constructed based on contrast learning through iterative training; A splicing module for splicing the prompt template matrix and the target pre-trained word vector matrix to obtain a spliced target sentence vector representation; inputting the target sentence vector representation into an encoding model to obtain a target sentence representation corresponding to the target text data; A classification module for mapping and classifying the target sentence representation using a linear classifier to obtain a label set corresponding to the target text data.

9. A computer device, comprising: A processor and a memory for storing processor-executable instructions, the processor executing the instructions to implement the steps of the method of any one of claims 1 to 7.

10. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images