Multi-label text classification method and device based on retrieval and readable medium

Abstract

The invention discloses a retrieval-based multi-label text classification method and device and a readable medium. The method comprises the following steps: training a teacher model by adopting a hybrid enhancement mode; inputting the candidate texts in the sample space into a trained teacher model, and establishing a knowledge vector library for retrieval; respectively inputting the training text into a trained teacher model and a trained student model to obtain a second embedded representation, a second logs feature, a third embedded representation and a third logs feature so as to construct a loss function of knowledge distillation based on a close constraint loss function and a cross entropy loss function, and training the student model; and inputting a to-be-classified text into the trained student model to obtain fourth embedded representations and fourth logs features, and retrieving k neighbor first embedded representations and corresponding k neighbor first logs features in the knowledge vector library according to the fourth embedded representations to perform enhanced fusion to obtain enhanced logs features so as to improve the text classification accuracy of the student model.

Description

Technical Field

[0001] The present invention relates to the field of text classification, and in particular to a retrieval-based multi-label text classification method, device and readable medium. Background Art

[0002] Multi-label text classification tasks are common text classification tasks, and when it comes to classification tasks, deep learning methods of artificial intelligence undoubtedly play a key role. By analyzing the input text, the system is able to automatically classify it into specific categories without human intervention. This technology has a revolutionary impact in many fields, including medicine, law, finance, etc. For the medical field, for example, when a patient describes symptoms, the system can accurately identify possible diseases or health problems, providing valuable reference for doctors. In the legal field, automatic text classification can help lawyers find relevant legal cases more quickly and accurately, and improve the efficiency of legal research.

[0003] As technology develops, deep learning models continue to grow, and their generalization and problem-solving capabilities have been greatly improved. The emergence of large-scale deep learning models enables computers to process large-scale, complex text data. This ability is not limited to English text, but also performs well in processing text in various languages. For example, in a multilingual environment, these models can automatically identify the language of the text and classify it accordingly, providing great convenience for information processing on a global scale.

[0004] However, in the industry, classification is usually only used as a means of control. If you rely on a large model, the consumption of hardware resources will be very exaggerated. The industry now widely uses three methods: pruning, quantization, and distillation to simplify the capabilities of large models into small models to reduce hardware resource consumption. Distillation is easier to implement and maintain during the development process in the industry, so it has received widespread attention. Even if the distillation framework can enable small models to learn the capabilities of large models to a certain extent, small models are still prone to making mistakes in judging "biased, strange, and new" problems. The main reason for these problems is that the number of parameters of the small model is insufficient compared to the large model, and the learning capacity and generalization are insufficient. Summary of the invention

[0005] In view of the technical problems mentioned above, the purpose of the embodiments of the present application is to propose a retrieval-based multi-label text classification method, device and readable medium to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, the present invention provides a retrieval-based multi-label text classification method, comprising the following steps:

[0007] Construct a teacher model and a student model, and train the teacher model using a hybrid enhancement method to obtain a trained teacher model;

[0008] Obtain candidate texts to form a sample space, input the candidate texts in the sample space into a trained teacher model, obtain a first embedding representation and a first logits feature, and establish a knowledge vector library for retrieval based on the first embedding representation and the first logits feature;

[0009] Obtain training text, input the training text into the trained teacher model to obtain a second embedding representation and a second logits feature, input the training text into the student model to obtain a third embedding representation and a third logits feature, combine the proximity constraint loss function of the second embedding representation and the third embedding representation and the cross entropy loss function of the second logits feature and the third logits feature to obtain a loss function of knowledge distillation, train the student model based on the loss function of knowledge distillation, and obtain a trained student model;

[0010] A text to be classified is obtained, and the text to be classified is input into a trained student model to obtain a fourth embedding representation and a fourth logits feature, and the first embedding representations of k nearest neighbors and the first logits features of the corresponding k nearest neighbors are retrieved in a knowledge vector library according to the fourth embedding representation, and the fourth logits feature is enhanced and fused according to the fourth embedding representation, the first embedding representations of the k nearest neighbors and the first logits features of the k nearest neighbors to obtain an enhanced logits feature, and a text classification result is obtained according to the enhanced logits feature.

[0011] Preferably, a hybrid enhancement method is used to train the teacher model to obtain a trained teacher model, specifically including:

[0012] Obtain training samples, which include training sets and test sets. Use the training samples to train the teacher model in the first stage. Use the cross entropy loss function in the first stage of training to obtain the teacher model trained in the first stage, and determine the classification sample distribution of the training set.

[0013] Construct an alpha weight distribution function based on the classification sample distribution of the training set;

[0014] The teacher model trained in the first stage is trained in the second stage using the training samples. The focal loss function is used for training in the second stage, and the alpha weight in the focal loss function is dynamically optimized using the alpha weight distribution function.

[0015] Preferably, constructing an alpha weight distribution function based on the classification sample distribution of the training set specifically includes:

[0016] Count the total number of samples in each category in the training set, calculate the ratio a of the total number of samples in each category to the total number of samples in the training set, and construct the classification sample distribution of the training set;

[0017] An alpha weight distribution function is constructed according to the proportion a of each category. In the alpha weight distribution function, the proportion a of each category and the alpha weight satisfy a nonlinear and monotonic mathematical mapping.

[0018] Preferably, constructing an alpha weight distribution function based on the classification sample distribution of the training set specifically includes:

[0019] Count the total number of samples in each category in the training set, calculate the ratio a of the total number of samples in each category to the total number of samples in the training set, and construct the classification sample distribution of the training set;

[0020] The teacher model trained in the first stage is tested using the test set to obtain the precision p and recall r of each category;

[0021] An alpha weight distribution function is constructed according to the proportion a, accuracy p and recall rate r of each category. The proportion a, accuracy p and recall rate r of each category in the alpha weight distribution function and the alpha weight satisfy a nonlinear and monotonic mathematical mapping.

[0022] Preferably, the proximity constraint loss function between the second embedding representation and the third embedding representation is the Euclidean distance between the second embedding representation and the third embedding representation.

[0023] Preferably, the formula of the close constraint loss function of the second embedding representation and the third embedding representation is as follows:

[0024]

[0025] Among them, A@B represents the cosine similarity between A and B, temperature represents the distillation temperature, and e T is the second embedding representation, e S is the third embedding representation, and kl-div(·) is the function corresponding to the kl divergence.

[0026] Preferably, the fourth logits feature is enhanced and fused according to the fourth embedding representation, the first embedding representation of the k nearest neighbors, and the first logits features of the k nearest neighbors to obtain an enhanced logits feature, specifically including:

[0027] The fourth embedding representation is denoted as e′ S , the fourth logits feature is denoted as l′ S , the first embedding representation of the k nearest neighbors is denoted as e k, the first logits feature of the k nearest neighbors is represented as l k , k∈K, the formula for enhanced fusion is:

[0028] score = softmax(E);

[0029] Where E is the neighbor matrix, and the kth element E[k] of E is: E[k] = e′ S ·e k ;

[0030] The calculation formula of the enhanced logits feature is as follows:

[0031] logits 增强 = l′ S +β*∑ k∈K score[k]*l k ;

[0032] Among them, β is a hyperparameter.

[0033] In a second aspect, the present invention provides a retrieval-based multi-label text classification device, comprising:

[0034] A teacher model training module is configured to construct a teacher model and a student model, and train the teacher model in a hybrid enhancement manner to obtain a trained teacher model;

[0035] A knowledge vector library building module is configured to obtain candidate texts to form a sample space, input the candidate texts in the sample space into a trained teacher model, obtain a first embedding representation and a first logits feature, and build a knowledge vector library for retrieval based on the first embedding representation and the first logits feature;

[0036] The student model training module is configured to obtain a training text, input the training text into a trained teacher model to obtain a second embedding representation and a second logits feature, input the training text into a student model to obtain a third embedding representation and a third logits feature, combine a proximity constraint loss function between the second embedding representation and the third embedding representation and a cross entropy loss function between the second logits feature and the third logits feature to obtain a loss function of knowledge distillation, train the student model based on the loss function of knowledge distillation, and obtain a trained student model;

[0037] The execution module is configured to obtain a text to be classified, input the text to be classified into a trained student model, obtain a fourth embedding representation and a fourth logits feature, retrieve the first embedding representations of k nearest neighbors and their corresponding first logits features in a knowledge vector library according to the fourth embedding representation, enhance and fuse the fourth logits feature according to the fourth embedding representation, the first embedding representations of the k nearest neighbors and the first logits features of the k nearest neighbors to obtain an enhanced logits feature, and obtain a text classification result according to the enhanced logits feature.

[0038] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The retrieval-based multi-label text classification method proposed in the present invention trains a student model by constructing a loss function based on the third embedding representation and the third logits feature, a close constraint loss function between the second embedding representation and the third embedding representation, and a cross entropy loss function between the second logits feature and the third logits feature. The close constraint loss function can make the hidden space of the student model close to the hidden space of the trained teacher model to facilitate subsequent retrieval tasks.

[0042] (2) The retrieval-based multi-label text classification method proposed in the present invention adopts a hybrid enhancement method to train the teacher model. During the training process of the teacher model, the alpha weight distribution function of the focal loss function is used to dynamically optimize the multi-order iterative model. It can perform enhanced constraints on unevenly distributed samples and difficult and easy samples, improve the performance in a small sample space, and retain the ability of the original sample space with good performance within an allowable range, thereby improving the robustness of the training.

[0043] (3) The retrieval-based multi-label text classification method proposed in the present invention solves the problem of insufficient learning ability of the student model for "biased, strange, and new" small sample spaces in engineering, and improves the text classification accuracy of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application may be applied;

[0046] Figure 2 A flowchart of a retrieval-based multi-label text classification method according to an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a retrieval-based multi-label text classification device according to an embodiment of the present application;

[0048] Figure 4 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Figure 1 An exemplary device architecture 100 is shown to which the retrieval-based multi-label text classification method or retrieval-based multi-label text classification device according to the embodiment of the present application can be applied.

[0051] like Figure 1 As shown, the device architecture 100 may include a terminal device 1 101, a terminal device 2 102, a terminal device 3 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device 1 101, the terminal device 2 102, the terminal device 3 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0052] The user can use the terminal device 101, the terminal device 2 102, and the terminal device 3 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications, such as data processing applications, file processing applications, etc., can be installed on the terminal device 101, the terminal device 2 102, and the terminal device 3 103.

[0053] Terminal device 1 101, terminal device 2 102, and terminal device 3 103 can be hardware or software. When terminal device 1 101, terminal device 2 102, and terminal device 3 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. When terminal device 1 101, terminal device 2 102, and terminal device 3 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0054] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal device 101, the terminal device 2 102, and the terminal device 3 103. The background data processing server may process the acquired files or data and generate processing results.

[0055] It should be noted that the retrieval-based multi-label text classification method provided in the embodiment of the present application can be executed by the server 105, and can also be executed by the terminal device 1 101, the terminal device 2 102, and the terminal device 3 103. Accordingly, the retrieval-based multi-label text classification device can be set in the server 105, and can also be set in the terminal device 1 101, the terminal device 2 102, and the terminal device 3 103.

[0056] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. In the case where the processed data does not need to be obtained remotely, the above device architecture may not include a network, but only a server or a terminal device.

[0057] Figure 2 A multi-label text classification method based on retrieval provided in an embodiment of the present application is shown, comprising the following steps:

[0058] S1, construct a teacher model and a student model, and train the teacher model using a hybrid enhancement method to obtain a trained teacher model.

[0059] In a specific embodiment, a hybrid enhancement method is used to train the teacher model to obtain a trained teacher model, which specifically includes:

[0060] Obtain training samples, which include training sets and test sets. Use the training samples to train the teacher model in the first stage. Use the cross entropy loss function in the first stage of training to obtain the teacher model trained in the first stage, and determine the classification sample distribution of the training set.

[0061] Construct an alpha weight distribution function based on the classification sample distribution of the training set;

[0062] The teacher model trained in the first stage is trained in the second stage using the training samples. The focal loss function is used for training in the second stage, and the alpha weight in the focal loss function is dynamically optimized using the alpha weight distribution function.

[0063] In a specific embodiment, constructing an alpha weight distribution function based on the classification sample distribution of a training set specifically includes:

[0064] Count the total number of samples in each category in the training set, calculate the ratio a of the total number of samples in each category to the total number of samples in the training set, and construct the classification sample distribution of the training set;

[0065] An alpha weight distribution function is constructed according to the proportion a of each category. In the alpha weight distribution function, the proportion a of each category and the alpha weight satisfy a nonlinear and monotonic mathematical mapping.

[0066] In a specific embodiment, constructing an alpha weight distribution function based on the classification sample distribution of a training set specifically includes:

[0067] Count the total number of samples in each category in the training set, calculate the ratio a of the total number of samples in each category to the total number of samples in the training set, and construct the classification sample distribution of the training set;

[0068] The teacher model trained in the first stage is tested using the test set to obtain the precision p and recall r of each category;

[0069] An alpha weight distribution function is constructed according to the proportion a, accuracy p and recall rate r of each category. The proportion a, accuracy p and recall rate r of each category in the alpha weight distribution function and the alpha weight satisfy a nonlinear and monotonic mathematical mapping.

[0070] Specifically, in text classification, the teacher model can adopt the RoBERTa large model. The student model can adopt any model adapted to the task, and the long short-term memory network (LSTM) is widely used in the industry. The student model in the embodiment of the present application adopts a model based on the LSTM architecture. Generally speaking, when training the model, a single Loss constraint model convergence will be adopted, and the Loss drops to the micro-oscillation stage or the accuracy does not increase. The embodiment of the present application uses a hybrid enhancement method to train the teacher model. The hybrid enhancement method refers to the training process, which divides the training into two stages. First, the CrossEntropyLoss (cross entropy loss function) is used in the training process of the first stage to evenly learn each category. After constraining the model to a good ability, the FocolLoss (focus loss function) is used in the training process of the second stage to perform enhanced constraints on unevenly distributed samples and difficult and easy samples. In the training process of the two stages, the training samples remain consistent.

[0071] The main reason for using hybrid enhancement in the training process of the teacher model is that if the focal loss function is used for constraint at the beginning of training, the uncertainty of the rule for constructing the alpha weight may lead to insufficient feature learning in the large sample space. The alpha weight is a hyperparameter in the focal loss function, which is designed by the specific task. Two design methods that have been proven to be effective in practice will be given later to dynamically optimize the alpha weight. The hybrid enhancement method will improve the robustness of training. First, the cross entropy loss function is used to balance the samples of different categories, then the learning rate is lowered, and the training is continued under the constraint of the focal loss function. This training method can improve the performance of the model in a small sample space, and can retain the ability of the sample space with good performance within the allowable range. This training method can achieve multi-layer cascading, and each time a new alpha weight distribution function is used to reconstruct the back propagation influence of the focal loss function. Specifically, after the cross entropy loss function is used for constraint training convergence, the performance of different categories obtained by evaluation can be used to construct the alpha weight distribution function of the focal loss function and train it, and then a new alpha weight distribution function is constructed based on the new evaluation, and training is continued until the desired performance distribution is achieved.

[0072] The embodiments of the present application explore and innovate the construction of the alpha weight distribution function, and the two most effective methods are:

[0073] 1. Perform nonlinear and monotonic mathematical mapping based on the distribution of classified samples in the training set (the specific formula needs to be adjusted according to the distribution);

[0074] The specific steps are as follows:

[0075] a) First, count the total number of samples in each category in the training set and calculate its proportion a in the entire training set. The proportion a of all categories constitutes the classification sample distribution of the training set;

[0076] b) According to the ratio a, an alpha weight distribution function for each category is designed, and the function can satisfy the characteristics of nonlinearity and monotonically decreasing, and is not limited to a specific function. For example: alpha_a = exp(1-a).

[0077] 2. According to the classification sample distribution of the training set, the accuracy distribution of the test, and the recall distribution of the test, the fusion is performed to obtain the fusion result, and nonlinear and monotonic mathematical mapping is performed based on the fusion result.

[0078] The specific steps are as follows:

[0079] a) Use the model to test on the test set to obtain the precision p and recall r of each category;

[0080] b) Determine the alpha weight distribution function of each category based on the three coefficients of the proportion a, precision p and recall r of each category. The function can satisfy the characteristics of nonlinearity and monotonically decreasing, and is not limited to a specific function. For example: alpha_a = exp(1-a*p*r).

[0081] S2, obtaining candidate texts to form a sample space, inputting the candidate texts in the sample space into the trained teacher model, obtaining a first embedding representation and a first logits feature, and establishing a knowledge vector library for retrieval based on the first embedding representation and the first logits feature.

[0082] Specifically, the knowledge vector library uses the faiss database, and the retrieval methods used include the FLAT algorithm or the HNSW algorithm. In other words, faiss can be used to quickly build a knowledge vector library. After obtaining the trained teacher model, the trained teacher model can be used to build a library for the sample space that you want to enhance. The purpose of building the library is to obtain the first embedding representation and the first logits feature obtained by the reasoning of the trained teacher model on the selected sample space. The first embedding representation belongs to embeddings, which represents the semantic understanding of the trained teacher model. Embeddings is the output of the embedding layer; the first logits feature belongs to logits, which represents the inference of the trained teacher model. Logits is the output of the final fully connected layer. Before passing through the softmax function layer or the sigmoid function layer, it is the unnormalized probability.

[0083] The first embedding representation and the first logits feature obtained by reasoning with the trained teacher model need to be stored in a format that is easy to retrieve. The embodiment of the present application uses the faiss database to build the database. As for the retrieval method, if an accurate match is desired, the FLAT algorithm is used, which is a brute force type of two-to-two matching algorithm. This retrieval method is relatively slow. If you want to retrieve faster but sacrifice a certain degree of accuracy, you can use the HNSW algorithm. Macroscopically speaking, the retrieval here is k-ANN (k nearest neighbor fuzzy search), which has nothing to do with the embeddings and logits of the vector library, and is mainly selected based on the desired effect. There are other retrieval and matching algorithms in the Faiss database, which will not be described here.

[0084] S3, obtain the training text, input the training text into the trained teacher model, obtain the second embedding representation and the second logits feature, input the training text into the student model, obtain the third embedding representation and the third logits feature, combine the proximity constraint loss function of the second embedding representation and the third embedding representation and the cross entropy loss function of the second logits feature and the third logits feature to obtain the loss function of knowledge distillation, train the student model based on the loss function of knowledge distillation, and obtain the trained student model.

[0085] In a specific embodiment, the proximity constraint loss function between the second embedding representation and the third embedding representation is the Euclidean distance between the second embedding representation and the third embedding representation.

[0086] In a specific embodiment, the formula of the close constraint loss function of the second embedding representation and the third embedding representation is as follows:

[0087]

[0088] Among them, A@B represents the cosine similarity between A and B, temperature represents the distillation temperature, and e T is the second embedding representation, e S is the third embedding representation, and kl-div(·) is the function corresponding to the kl divergence.

[0089] Specifically, in the traditional knowledge distillation process, the loss function is composed of two supervisory constraints, one is the soft label cross entropy loss function between the second logits feature output by the student model and the third logits feature of the trained teacher model, and the other is the hard label cross entropy loss function between the text classification result output by the student model and the true label. The embodiment of the present application modifies the hard label cross entropy loss function into a close constraint of the embeddings output by the student model to the embeddings output by the trained teacher model, that is, the close constraint loss function of the second embedding representation and the third embedding representation and the cross entropy loss function of the second logits feature and the third logits feature are added to obtain the loss function of knowledge distillation. The essence of this close constraint is to hope that the hidden space of the student model can be close to the hidden space of the trained teacher model to facilitate subsequent retrieval tasks. There are many ways to calculate the close constraint loss function. The simplest way is to use the Euclidean distance to calculate the distance difference between the embeddings output by the trained teacher model and the embeddings output by the student model. Another way is to use the idea of ​​contrastive learning, take the embeddings of the teacher model as the supervised object, form the Supervised Contrast Matrix, and then use KL divergence to calculate the loss. The specific calculation process is as follows:

[0090] Input a batch of training texts into the trained teacher model to obtain a batch of embeddings, which are recorded as the second embedding representation e T , input a batch of training texts into the student model to obtain a batch of embeddings, recorded as the third embedding representation e S . Then perform the following KL divergence calculation:

[0091] KL Divergence Loss =kl-div(softmax(e T @e T / temperature),softmax(e T @e S /

[0092] temperature);

[0093] The @ symbol indicates that the embeddings of each sample in two matrices (dimension is batch size multiplied by hidden layer dimension) are calculated for cosine similarity, so the final dimension is batch size multiplied by batch size; temperature indicates distillation temperature. Therefore, the student model can learn more information. It is worth noting that if the teacher model adopts Bert architecture, the embeddings are the embeddings of the [CLS] position instead of all the embeddings of the entire sentence.

[0094] S4, obtaining the text to be classified, inputting the text to be classified into the trained student model, obtaining a fourth embedding representation and a fourth logits feature, retrieving the first embedding representations of k neighbors and their corresponding first logits features of the k neighbors in the knowledge vector library according to the fourth embedding representation, enhancing and fusion the fourth logits feature according to the fourth embedding representation, the first embedding representations of the k neighbors and the first logits features of the k neighbors, obtaining an enhanced logits feature, and obtaining a text classification result according to the enhanced logits feature.

[0095] In a specific embodiment, the fourth logits feature is enhanced and fused according to the fourth embedding representation, the first embedding representation of the k nearest neighbors, and the first logits features of the k nearest neighbors to obtain an enhanced logits feature, specifically including:

[0096] The fourth embedding representation is denoted as e′ S , the fourth logits feature is denoted as l′ S , the first embedding representation of the k nearest neighbors is denoted as e k , the first logits feature of the k nearest neighbors is represented as l k , k∈K, the formula for enhanced fusion is:

[0097] score[k] = softmax(E[k]);

[0098] Where E is the neighbor matrix, and the kth element E[k] of E is: E[k] = e′ S ·e k ;

[0099] The calculation formula of the enhanced logits feature is as follows:

[0100] logits 增强 = l′ S +β*∑ k∈K score[k]*l k ;

[0101] Among them, β is a hyperparameter.

[0102] Specifically, the embeddings inferred by the trained student model are used for enhanced reasoning for retrieval enhancement during the reasoning process. The training of the student model and the construction of the knowledge vector library based on the trained teacher model are completed through steps S1-S3. When performing reasoning, an enhancement step is required. In an embodiment of the present application, the faiss database will be used to read the vector library and retrieve the vector library. Simply put, after the trained student model infers the embedding, according to the library construction algorithm, k adjacent embeddings are retrieved from the knowledge vector library, and through the mechanism of attention, the similarity between the embeddings output by the trained student model and the k adjacent embeddings retrieved from the knowledge vector library is obtained, and then the similarity is used as a weight to weight the enhanced logits feature, and finally the text classification result is output according to the enhanced logits feature.

[0103] The retrieval-based multi-label text classification method proposed in the embodiment of the present application was tested with the traditional direct classification method. The experimental data is shown in Table 1. The trained student model obtained by the retrieval-based multi-label text classification method proposed in the embodiment of the present application can improve the performance of the basic pipeline by 2.73% F1 value, 4.97% recall rate, and 0.35% accuracy. The data comes from the classification data set of the anorectal department in the medical field.

[0104] Table 1

[0105]

[0106]

[0107] The above steps S1-S4 do not necessarily represent the order between the steps, but are represented by step symbols, and the order between the steps can be adjusted.

[0108] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a retrieval-based multi-label text classification device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] The present application embodiment provides a retrieval-based multi-label text classification device, including:

[0110] The teacher model training module 1 is configured to construct a teacher model and a student model, and train the teacher model using a hybrid enhancement method to obtain a trained teacher model;

[0111] The knowledge vector library building module 2 is configured to obtain a sample space composed of candidate texts, input the candidate texts in the sample space into a trained teacher model, obtain a first embedding representation and a first logits feature, and build a knowledge vector library for retrieval according to the first embedding representation and the first logits feature;

[0112] The student model training module 3 is configured to obtain a training text, input the training text into the trained teacher model, obtain a second embedding representation and a second logits feature, input the training text into the student model, obtain a third embedding representation and a third logits feature, combine the proximity constraint loss function of the second embedding representation and the third embedding representation and the cross entropy loss function of the second logits feature and the third logits feature to obtain a loss function of knowledge distillation, train the student model based on the loss function of knowledge distillation, and obtain a trained student model;

[0113] Execution module 4 is configured to obtain a text to be classified, input the text to be classified into a trained student model, obtain a fourth embedding representation and a fourth logits feature, retrieve the first embedding representations of k neighbors and their corresponding first logits features of the k neighbors in the knowledge vector library according to the fourth embedding representation, enhance and fuse the fourth logits feature according to the fourth embedding representation, the first embedding representations of the k neighbors and the first logits features of the k neighbors to obtain an enhanced logits feature, and obtain a text classification result according to the enhanced logits feature.

[0114] Reference below Figure 4 , which shows an electronic device (eg Figure 1 A schematic diagram of the structure of a computer device 400 (a server or terminal device as shown). Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 4 As shown, the computer device 400 includes a central processing unit (CPU) 401 and a graphics processing unit (GPU) 402, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 403 or a program loaded from a storage part 409 to a random access memory (RAM) 404. In the RAM 404, various programs and data required for the operation of the computer device 400 are also stored. The CPU 401, GPU 402, ROM 403 and RAM 404 are connected to each other through a bus 405. An input / output (I / O) interface 406 is also connected to the bus 405.

[0116] The following components are connected to the I / O interface 406: an input section 407 including a keyboard, a mouse, etc.; an output section 408 including a display such as a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 409 including a hard disk, etc.; and a communication section 410 including a network interface card such as a LAN card, a modem, etc. The communication section 410 performs communication processing via a network such as the Internet. A drive 411 can also be connected to the I / O interface 406 as needed. A removable medium 412, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 411 as needed so that a computer program read therefrom is installed into the storage section 409 as needed.

[0117] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 410, and / or installed from a removable medium 412. When the computer program is executed by a central processing unit (CPU) 401 and a graphics processing unit (GPU) 402, the above-mentioned functions defined in the method of the present application are executed.

[0118] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor device, device or component, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution device, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution device, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0119] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The modules involved in the embodiments described in this application may be implemented by software or hardware, and the modules described may also be set in a processor.

[0122] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: constructs a teacher model and a student model, trains the teacher model using a hybrid enhancement method, and obtains a trained teacher model; obtains a sample space composed of candidate texts, inputs the candidate texts in the sample space into the trained teacher model, obtains a first embedding representation and a first logits feature, and establishes a knowledge vector library for retrieval based on the first embedding representation and the first logits feature; obtains training text, inputs the training text into the trained teacher model, obtains a second embedding representation and a second logits feature, inputs the training text into the student model, obtains a third embedding representation and a third logits feature, and associates the second embedding representation with the third embedding representation. The near constraint loss function and the cross entropy loss function of the second logits feature and the third logits feature are combined to obtain the loss function of knowledge distillation, and the student model is trained based on the loss function of knowledge distillation to obtain a trained student model; the text to be classified is obtained, and the text to be classified is input into the trained student model to obtain a fourth embedding representation and a fourth logits feature, and the first embedding representations of k neighbors and their corresponding first logits features of the k neighbors are retrieved from the knowledge vector library according to the fourth embedding representation, and the fourth logits feature is enhanced and fused according to the fourth embedding representation, the first embedding representations of the k neighbors and the first logits features of the k neighbors to obtain an enhanced logits feature, and a text classification result is obtained according to the enhanced logits feature.

[0123] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A retrieval-based multi-label text classification method, characterized in that: The following steps are involved: Constructing a teacher model and a student model, and training the teacher model using a hybrid enhancement method to obtain a trained teacher model; Acquire candidate texts to form a sample space, input the candidate texts in the sample space into the trained teacher model, obtain a first embedding representation and a first logits feature, and establish a knowledge vector library for retrieval based on the first embedding representation and the first logits feature; Obtaining a training text, inputting the training text into the trained teacher model to obtain a second embedding representation and a second logits feature, inputting the training text into the student model to obtain a third embedding representation and a third logits feature, combining a proximity constraint loss function between the second embedding representation and the third embedding representation and a cross entropy loss function between the second logits feature and the third logits feature to obtain a loss function for knowledge distillation, and training the student model based on the loss function for knowledge distillation to obtain a trained student model; A text to be classified is obtained, and the text to be classified is input into the trained student model to obtain a fourth embedding representation and a fourth logits feature, and the first embedding representations of k neighbors and the first logits features of their corresponding k neighbors are retrieved from the knowledge vector library according to the fourth embedding representation, and the fourth logits feature is enhanced and fused according to the fourth embedding representation, the first embedding representations of the k neighbors, and the first logits features of the k neighbors to obtain an enhanced logits feature, and a text classification result is obtained according to the enhanced logits feature.

2. The retrieval-based multi-label text classification method according to claim 1 is characterized in that: The method of training the teacher model in a hybrid enhancement manner to obtain a trained teacher model specifically includes: Obtain training samples, the training samples including a training set and a test set, use the training samples to perform a first-stage training on the teacher model, use a cross entropy loss function in the first-stage training process to obtain the teacher model trained in the first stage, and determine the classification sample distribution of the training set; Constructing an alpha weight distribution function based on the classification sample distribution of the training set; The teacher model trained in the first stage is trained in the second stage using the training samples, and the focal loss function is used for training during the second stage of training, wherein the alpha weight in the focal loss function is dynamically optimized using the alpha weight distribution function.

3. The retrieval-based multi-label text classification method according to claim 2 is characterized in that: The constructing of the alpha weight distribution function based on the classified sample distribution of the training set specifically includes: Counting the total number of samples in each category in the training set, calculating the ratio a of the total number of samples in each category to the total number of samples in the training set, and forming a classification sample distribution of the training set; The alpha weight distribution function is constructed according to the proportion a of each category, and the proportion a of each category in the alpha weight distribution function and the alpha weight satisfy a nonlinear and monotonic mathematical mapping.

4. The retrieval-based multi-label text classification method according to claim 2 is characterized in that: The constructing of the alpha weight distribution function based on the classified sample distribution of the training set specifically includes: Counting the total number of samples in each category in the training set, calculating the ratio a of the total number of samples in each category to the total number of samples in the training set, and forming a classification sample distribution of the training set; The teacher model trained in the first stage is tested using the test set to obtain the precision p and recall r of each category; The alpha weight distribution function is constructed according to the proportion a, accuracy p and recall r of each category, and the proportion a, accuracy p and recall r of each category in the alpha weight distribution function and the alpha weight satisfy a nonlinear and monotonic mathematical mapping.

5. The retrieval-based multi-label text classification method according to claim 1 is characterized in that: The proximity constraint loss function between the second embedding representation and the third embedding representation is the Euclidean distance between the second embedding representation and the third embedding representation.

6. The retrieval-based multi-label text classification method according to claim 1 is characterized in that: The formula of the close constraint loss function of the second embedding representation and the third embedding representation is as follows: Among them, A@B represents the cosine similarity between A and B, temperature represents the distillation temperature, and e T For the second embedding representation, e S is the third embedding representation, and kl-div(·) is the function corresponding to the kl divergence.

7. The retrieval-based multi-label text classification method according to claim 1 is characterized in that: The step of enhancing and fusing the fourth logits feature according to the fourth embedding representation, the first embedding representations of the k nearest neighbors, and the first logits features of the k nearest neighbors to obtain an enhanced logits feature specifically includes: The fourth embedded representation is denoted as e′ S , the fourth logits feature is denoted as l′ S , the first embedding representation of k nearest neighbors is denoted as e k , the first logits feature of the k nearest neighbors is represented as l k , k∈K, the formula for enhanced fusion is: score[k] = softmax(E[k]); Where E is the neighbor matrix, and the kth element E[k] of E is: E[k] = e′ S ·e k ; The calculation formula of the enhanced logits feature is as follows: logits 增强 =l′ S +β*∑ k∈K score[k]*l k ; Among them, β is a hyperparameter.

8. A retrieval-based multi-label text classification device, characterized in that: include: A teacher model training module is configured to construct a teacher model and a student model, and train the teacher model in a hybrid enhancement manner to obtain a trained teacher model; A knowledge vector library building module is configured to obtain a sample space composed of candidate texts, input the candidate texts in the sample space into the trained teacher model, obtain a first embedding representation and a first logits feature, and build a knowledge vector library for retrieval according to the first embedding representation and the first logits feature; A student model training module is configured to obtain a training text, input the training text into the trained teacher model to obtain a second embedding representation and a second logits feature, input the training text into the student model to obtain a third embedding representation and a third logits feature, combine a proximity constraint loss function of the second embedding representation and the third embedding representation and a cross entropy loss function of the second logits feature and the third logits feature to obtain a loss function of knowledge distillation, and train the student model based on the loss function of knowledge distillation to obtain a trained student model; The execution module is configured to obtain a text to be classified, input the text to be classified into the trained student model, obtain a fourth embedding representation and a fourth logits feature, retrieve the first embedding representations of k neighbors and the first logits features of their corresponding k neighbors in the knowledge vector library according to the fourth embedding representation, enhance and fuse the fourth logits feature according to the fourth embedding representation, the first embedding representations of the k neighbors and the first logits features of the k neighbors to obtain an enhanced logits feature, and obtain a text classification result according to the enhanced logits feature.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images