Method, device and storage medium for training multi-label classification model

Abstract

The application discloses a kind of multi-label classification model training method, device, equipment and storage medium, comprising: the text vector of the training sample obtained is respectively input into the first training network and the second training network of the training model, and the first probability and the second probability of training sample are obtained;According to the first probability and the second probability, the first loss value and the second loss value corresponding to the first training network are respectively calculated;According to the first loss value, the second loss value, the first weight value of the first loss value of preset and the second weight value of the second loss value of preset, determine the third loss value of total model;If the third loss value meets preset condition, the training model is used as multi-label classification model;If the third loss value does not meet preset condition, according to the third loss value, the parameter of the training model is adjusted, the next training is carried out, reduces irrelevant or lower correlation label to be calculated, accelerates model training speed, improves the accuracy of model classification.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically relating to a training method, apparatus, device, and storage medium for a multi-label classification model. Background Technology

[0002] Multi-label refers to a sample that may belong to multiple classes simultaneously, meaning it has multiple labels. For example, a size L winter coat would have at least two labels: size: L, and type: winter clothing. Training a machine learning model using multi-labeled samples yields a multi-label classification model. This model can identify a target, and the output can be a vector representing the category to which the target belongs.

[0003] Currently, there are many multi-label learning algorithms, which can be divided into two main categories based on the problem-solving perspective: one is based on problem transformation, and the other is based on algorithm applicability.

[0004] One approach based on problem transformation is to transform the problem data to enable the use of existing algorithms. For example, algorithms that disregard correlations treat each label in a multi-label dataset as a single label, applying common classification algorithms to each label. Specifically, in traditional machine learning models, binary classification is performed for each label class, using algorithms such as SVM, DT, Native Bayes, and XGBoost. In deep learning, a text classification model (e.g., textCNN, textRNN) is trained for each class. Besides treating labels separately, there is also the approach of treating labels uniformly (Label Powerset). In this regard, we transform the problem into a multi-class problem, where a multi-class classifier is trained on all unique label combinations found in the training data.

[0005] The second approach, based on algorithm applicability, refers to extending a specific algorithm to handle multi-label data, improving the algorithm, and making it suitable for different types of data. Examples include MLkNN, a multi-label version of KNN, and Rank-SVM, a multi-label version of SVM. In deep learning, this often involves modifying the output layer of a multi-classification model to make it suitable for multi-label classification.

[0006] However, the correlation between labels and text often plays a very important role during model training. Most existing models ignore this relationship and often calculate the probability of labels that are unrelated to the text or have low correlation. Obviously, these probabilities are unnecessary, which not only increases the training error but also reduces the accuracy of model classification. Summary of the Invention

[0007] The main objective of this invention is to provide a training method, apparatus, device, and storage medium for a multi-label classification model, in order to solve the technical problems of large training errors and low classification accuracy of existing multi-label classification models.

[0008] To address the above problems, this invention provides a training method for a multi-label classification model, comprising:

[0009] Obtain the text vectors of the training samples;

[0010] The text vector is input into the first training network and the second training network of the current training model to obtain the first probability and the second probability of the training sample; wherein, the first training network is a network without an attention mechanism, and the second training network is a network with an attention mechanism.

[0011] Based on the first probability and the actual probability of the training sample, calculate the first loss value corresponding to the first training network; and based on the second probability and the actual probability of the training sample, calculate the second loss value corresponding to the second training network.

[0012] The third loss value of the overall model is determined based on the first loss value, the second loss value, the preset first weight value of the first loss value, and the preset second weight value of the second loss value.

[0013] If the third loss value meets the preset conditions, the current training model will be used as the multi-label classification model.

[0014] If the third loss value does not meet the preset conditions, the parameters of the current training model are adjusted according to the third loss value until the third loss value obtained by the next training model meets the preset conditions.

[0015] Furthermore, in the training method of the multi-label classification model described above, determining the third loss value of the overall model based on the first loss value, the second loss value, a preset first weight value for the first loss value, and a preset second weight value for the second loss value includes:

[0016] Determine a first product of the first loss value and the first weight value, and determine a second product of the second loss value and the second weight value;

[0017] The sum of the first product and the second product is taken as the third loss value.

[0018] Furthermore, in the training method of the multi-label classification model described above, obtaining the text vector of the training samples includes:

[0019] The text vector is obtained by segmenting the text using Jieba and inputting it into the embedding layer of the model being trained.

[0020] Furthermore, in the training method of the multi-label classification model described above, the first training network includes an LSTM network or a GRU network.

[0021] The second training network includes a dynamic convolutional network.

[0022] Furthermore, in the training method of the multi-label classification model described above, the dynamic convolutional network includes a span-based dynamic convolutional network.

[0023] The present invention also provides a training device for a multi-label classification model, comprising:

[0024] The acquisition module is used to acquire the text vectors of the training samples;

[0025] The training module is used to input the text vector into the first training network and the second training network of the current training model to obtain the first probability and the second probability of the training sample; wherein, the first training network is a network without an attention mechanism, and the second training network is a network with an attention mechanism.

[0026] The first determining module is configured to calculate a first loss value corresponding to the first training network based on the first probability and the actual probability of the training sample; and to calculate a second loss value corresponding to the second training network based on the second probability and the actual probability of the training sample.

[0027] The second determining module is used to determine the third loss value of the total model based on the first loss value, the second loss value, a preset first weight value of the first loss value, and a preset second weight value of the second loss value.

[0028] The detection module is used to use the current training model as the multi-label classification model if the third loss value meets the preset conditions; if the third loss value does not meet the preset conditions, the module adjusts the parameters of the current training model according to the third loss value until the third loss value obtained by the next training model meets the preset conditions.

[0029] Furthermore, in the training device for the multi-label classification model described above, the second determining module is specifically used for:

[0030] Determine a first product of the first loss value and the first weight value, and determine a second product of the second loss value and the second weight value;

[0031] The sum of the first product and the second product is taken as the third loss value.

[0032] Furthermore, in the training device for the multi-label classification model described above, the acquisition module is specifically used for:

[0033] The text vector is obtained by segmenting the text using Jieba and inputting it into the embedding layer of the model being trained.

[0034] The present invention also provides a training device for a multi-label classification model, including a memory and a processor;

[0035] The memory stores a computer program that, when executed by a processor, implements the steps of the training method for the multi-label classification model as described in any of the preceding claims.

[0036] The present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for the multi-label classification model as described in any of the preceding claims.

[0037] Compared with the prior art, one or more embodiments of the above solutions may have the following advantages or beneficial effects:

[0038] The training method, apparatus, device, and storage medium for the multi-label classification model of the present invention, after obtaining the text vectors of the training samples, trains the model through a first training network without an attention mechanism to obtain a first loss value for the first training network. Then, by using a second training network with an attention mechanism, the model's representational power is improved by aggregating multiple convolutional kernels with attention without increasing network depth or width, and a second loss value for the second training network is obtained. Finally, a third loss value for the overall model is obtained based on the first and second loss values. The model is then trained based on the third loss value, reducing the calculation of irrelevant or low-relevance labels, accelerating model training, and improving the accuracy of model classification.

[0039] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0041] Figure 1 A flowchart illustrating an embodiment of the training method for the multi-label classification model of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of a training device embodiment for the multi-label classification model of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of an embodiment of the training device for the multi-label classification model of the present invention. Detailed Implementation

[0044] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples, so that the process of how the present invention uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly. It should be noted that, as long as there is no conflict, the various embodiments and features in the various embodiments of the present invention can be combined with each other, and the resulting technical solutions are all within the protection scope of the present invention.

[0045] In related technologies, the ML-KNN algorithm can be used for classification, as detailed below:

[0046] The main idea of ​​the ML-KNN algorithm is that for each new instance, the k nearest instances (the k instances with the smallest distance to it in the feature space) can be obtained first, then the label set of these instances can be obtained, and finally the label set of the new instance can be determined by the maximum a posteriori probability criterion.

[0047] First, we can use formula (1) to obtain the label information of the corresponding k nearest neighbor instances:

[0048]

[0049] Here, C x It is a 1*n row vector with C elements. x (1) refers to how many of the k nearest neighbors of label 1 x have this label.

[0050] For a new instance t, first obtain its k nearest neighbor index set N(t), and define the event. For t, there is label 1, event Define an event for t (without label 1). (j∈{0,1,…,k}) means that for label l, j out of k nearest neighbors contain this label. Then, based on vector C... t It can be obtained through the maximum a posteriori probability criterion and the Bayesian criterion, as shown in formula (2):

[0051]

[0052] Among them, y t (1) is the result we are looking for, which indicates whether the t instance has the l tag.

[0053] in The prior probability of whether label t has label l can be calculated by dividing the number of times label l appears in the entire training set by the total number of labels. See formula (3) for reference.

[0054]

[0055] That is, the number of vectors with the label l in the sample divided by the total number of vectors.

[0056] Posterior probability It can be calculated according to formula (4):

[0057]

[0058] Where, j in C[j] is equal to C t (1), that is, the number of labels l among the k nearest neighbors of t.

[0059] C[δ]+1, n∈{1,2,...,m} means: if x n The number of k-nearest neighbors with label l is δ, and x n If label 1 is given, then c[δ]+1, n∈{1,2,…,m}.

[0060] Then C[j] represents the number of vectors among all vectors that have j 1-labeled k-nearest neighbors and l-labeled themselves.

[0061] thus, This is the total number of vectors with label l and k nearest neighbors with labels l, ranging from 0 to m.

[0062] work out Then, we only need to see which case in b = {0, 1} maximizes this product value. If it is maximized when b = 1, then vector t has the label l; otherwise, it does not.

[0063] However, the algorithm has high computational and spatial complexity, low prediction accuracy for rare categories when samples are imbalanced, and poor interpretability because it does not consider the correlation between labels and text, and cannot provide rules like decision trees.

[0064] The TextCNN algorithm can also be used for classification in related technologies, as detailed below:

[0065] The structure of TextCNN:

[0066] Embedding layer: TextCNN uses pre-trained word vectors as the embedding layer. For all words in the dataset, since each word can be represented as a vector, we can obtain an embedding matrix MM, where each row of MM is a word vector.

[0067] Convolutional and pooling layers: For each word, along with the sentence embedding matrix M, a word vector can be obtained. Assume the word vector has d dimensions. Then, for this sentence, we can obtain an s x d matrix A ∈ Rs × d. Convolutional neural networks are used to extract features. One-dimensional convolution can be used; the difference between text convolution and image convolution is that convolution is performed only in one direction (vertical) of the text sequence, and the width of the convolution kernel is fixed to the dimension d of the word vector. The height is a hyperparameter that can be set. Convolution operations are performed on each possible window of the sentence words to obtain feature maps.

[0068] Pooling: Convolutional kernels of different sizes produce feature maps of different sizes. Therefore, we use a pooling function on each feature map to make them have the same dimension.

[0069] Softmax layer: We concatenate the results of max-pooling and feed them into softmax to obtain the probability of each category, such as the probability of label 1 and the probability of label -1.

[0070] However, the algorithm does not consider the relevance between labels and text, so the model is not very interpretable. When tuning the model, it is difficult to adjust specific features based on the training results, because there is no concept of feature importance in textCNN similar to that in GBDT model, so it is difficult to evaluate the importance of each feature.

[0071] Therefore, existing algorithms not only increase training error but also reduce the accuracy of model classification. To address these technical problems, this invention provides the following technical solution:

[0072] Definition: LSTM, Long Short-Term Memory Network, has shown good performance in processing sequential text tasks. It can well represent the global logical information and the complex temporal relationships between information in the input text.

[0073] CNN, or Convolutional Neural Network, is excellent for extracting local feature information of the central words in a sentence when used for text tasks.

[0074] Figure 1 A flowchart illustrating an embodiment of the training method for the multi-label classification model of the present invention is shown below. Figure 1 As shown, the training method for the multi-label classification model in this embodiment may specifically include the following steps:

[0075] 100. Obtain the text vectors of the training samples;

[0076] In a specific implementation, text can be segmented using Jieba and input into the embedding layer of the model being trained to obtain the text vector.

[0077] 101. Input the text vector into the first training network and the second training network of the current training model respectively to obtain the first probability and the second probability of the training sample;

[0078] In this configuration, the first training network is a network without an attention mechanism, while the second training network is a network with an attention mechanism. For example, the first training network may include an LSTM network or a GRU network; the second training network may include a dynamic convolutional network. The dynamic convolutional network may include a span-based dynamic convolutional network.

[0079] Specifically, taking LSTM as an example, after obtaining the text vector of the training sample, the text vector of the training sample can be input into the LSTM layer and the dynamic convolutional layer respectively, and then the classification probability is obtained through the softmax layer.

[0080] 102. Calculate the first loss value corresponding to the first training network based on the first probability and the actual probability of the training sample; and calculate the second loss value corresponding to the second training network based on the second probability and the actual probability of the training sample.

[0081] 103. Determine the third loss value of the overall model based on the first loss value, the second loss value, the preset first weight value of the first loss value, and the preset second weight value of the second loss value;

[0082] Specifically, a first product of the first loss value and the first weight value can be determined, and a second product of the second loss value and the second weight value can be determined; the sum of the first product and the second product is taken as the third loss value.

[0083] 104. Check whether the third loss value meets the preset conditions; if yes, proceed to step 105; if no, proceed to step 106.

[0084] 105. Use the current training model as the multi-label classification model;

[0085] If the third loss value meets a preset condition, the trained model for that iteration is used as the multi-label classification model. The preset condition can be that the third loss value is less than a preset threshold.

[0086] 106. Adjust the parameters of the current training model according to the third loss value, and return to step 100.

[0087] If the third loss value does not meet the preset conditions, the parameters of the current training model are adjusted according to the third loss value, and then the process returns to step 100 to continue training until the third loss value obtained by the next training model meets the preset conditions.

[0088] The training method of the multi-label classification model in this embodiment, after obtaining the text vectors of the training samples, trains the model through a first training network without the attention mechanism, obtaining a first loss value for the first training network. Then, by using a second training network with the attention mechanism, the model's representational power is improved by aggregating multiple convolutional kernels with attention without increasing network depth or width, resulting in a second loss value for the second training network. Finally, a third loss value for the overall model is obtained based on the first and second loss values. The model is then trained based on this third loss value, reducing the calculation of irrelevant or low-relevance labels, accelerating model training, and improving classification accuracy.

[0089] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the method.

[0090] Figure 2 This is a schematic diagram of the structure of an embodiment of the training device for the multi-label classification model of the present invention, as shown below. Figure 2 As shown, the training device for the multi-label classification model in this embodiment may include an acquisition module 20, a training module 21, a first determination module 22, a second determination module 23, and a detection module 24.

[0091] Module 20 is used to acquire the text vectors of the training samples;

[0092] Specifically, the text vector can be obtained by segmenting the text using Jieba and inputting it into the embedding layer of the model being trained.

[0093] Training module 21 is used to input the text vector into the first training network and the second training network of the current training model to obtain the first probability and the second probability of the training sample; wherein, the first training network is a network without an attention mechanism, and the second training network is a network with an attention mechanism.

[0094] The first determining module 22 is configured to calculate a first loss value corresponding to the first training network based on the first probability and the actual probability of the training sample; and to calculate a second loss value corresponding to the second training network based on the second probability and the actual probability of the training sample.

[0095] The first training network includes an LSTM network or a GRU network; the second training network includes a dynamic convolutional network. The dynamic convolutional network includes a span-based dynamic convolutional network.

[0096] The second determining module 23 is used to determine the third loss value of the total model based on the first loss value, the second loss value, a preset first weight value of the first loss value, and a preset second weight value of the second loss value.

[0097] Specifically, a first product of the first loss value and the first weight value can be determined, and a second product of the second loss value and the second weight value can be determined; the sum of the first product and the second product is taken as the third loss value.

[0098] The detection module 24 is used to use the current training model as the multi-label classification model if the third loss value meets the preset conditions; if the third loss value does not meet the preset conditions, the parameters of the current training model are adjusted according to the third loss value until the third loss value obtained by the next training model meets the preset conditions.

[0099] The training device for the multi-label classification model in this embodiment, after obtaining the text vectors of the training samples, trains the model using a first training network without an attention mechanism, resulting in a first loss value for the first training network. Then, by using a second training network with an attention mechanism, the model's representational power is improved by aggregating multiple convolutional kernels with attention, without increasing network depth or width, resulting in a second loss value for the second training network. Finally, a third loss value for the overall model is obtained based on the first and second loss values. The model is then trained based on this third loss value, reducing the calculation of irrelevant or low-relevance labels, accelerating model training, and improving classification accuracy.

[0100] The apparatus in the above embodiments is used to implement the corresponding methods in the foregoing embodiments. The specific implementation scheme can be found in the methods described in the foregoing embodiments and the relevant descriptions in the method embodiments. It also has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0101] Figure 3 This is a schematic diagram of the structure of an embodiment of the training device for the multi-label classification model of the present invention, as shown below. Figure 3As shown, the training device for the multi-label classification model in this embodiment may include a memory 30 and a processor 31;

[0102] The memory 30 stores a computer program, which, when executed by the processor 31, implements the steps of the method for dynamically weighting the loss function as described in the above embodiments.

[0103] This invention provides a storage medium storing a computer program. When the computer program is executed by a controller, it implements the steps of the training method for the multi-label classification model described above.

[0104] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0105] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0106] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0107] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0108] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0109] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0110] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0111] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] While the embodiments disclosed in this invention are as described above, the content is merely for the purpose of facilitating understanding of the invention and is not intended to limit the invention. Any person skilled in the art to which this invention pertains may make any modifications and changes in form and detail of the implementation without departing from the spirit and scope disclosed herein; however, the scope of protection of this invention shall still be determined by the scope defined in the appended claims.

Claims

1. A method for training a multi-label classification model, the method comprising: The method comprises the following steps: obtaining a text vector of a training sample; inputting the text vector into a first training network and a second training network of a current training model respectively to obtain a first probability and a second probability of the training sample; the first training network is a network without attention mechanism, and the second training network is a network with attention mechanism; calculating a first loss value corresponding to the first training network according to the first probability and an actual probability of the training sample, and calculating a second loss value corresponding to the second training network according to the second probability and the actual probability of the training sample; determining a third loss value of a total model according to the first loss value, the second loss value, a preset first weight value of the first loss value and a preset second weight value of the second loss value; if the third loss value meets a preset condition, taking the current training model as the multi-label classification model; if the third loss value does not meet the preset condition, adjusting parameters of the current training model according to the third loss value until the third loss value of a next training model meets the preset condition. 2.The method of Claim 1, wherein, The method for determining the third loss value of the total model according to the first loss value, the second loss value, the preset first weight value of the first loss value and the preset second weight value of the second loss value comprises the following steps: determining a first product of the first loss value and the first weight value, and determining a second product of the second loss value and the second weight value; taking a sum of the first product and the second product as the third loss value. 3.The method of Claim 1, wherein, The method for obtaining the text vector of the training sample comprises the following steps: performing word segmentation on the text by using Jieba, and inputting the text into an embedding layer of the current training model to obtain the text vector. 4.The method of Claim 1, wherein, The first training network comprises an LSTM network or a GRU network. The second training network comprises a dynamic convolution network. 5.The method of Claim 4, wherein, The dynamic convolution network comprises a span-based dynamic convolution network. 6.A device for training a multi-label classification model, comprising: The method comprises the following steps: an obtaining module is configured to obtain a text vector of a training sample; a training module is configured to input the text vector into a first training network and a second training network of a current training model respectively to obtain a first probability and a second probability of the training sample; the first training network is a network without attention mechanism, and the second training network is a network with attention mechanism; a first determining module is configured to calculate a first loss value corresponding to the first training network according to the first probability and an actual probability of the training sample, and calculate a second loss value corresponding to the second training network according to the second probability and the actual probability of the training sample; a second determining module is configured to determine a third loss value of a total model according to the first loss value, the second loss value, a preset first weight value of the first loss value and a preset second weight value of the second loss value. The detection module is configured to: if the third loss value meets a preset condition, take the current training model as the multi-label classification model; and if the third loss value does not meet the preset condition, perform parameter adjustment on the current training model according to the third loss value until the third loss value obtained by a next training model meets the preset condition.

7. The training apparatus of a multi-label classification model according to claim 6, wherein, The second determination module is specifically configured to: determine a first product of the first loss value and the first weight value, and determine a second product of the second loss value and the second weight value; take a sum of the first product and the second product as the third loss value. 8.The device of claim 6, wherein, The acquisition module is specifically configured to: perform text segmentation on the text by using Jieba, and input the segmented text into an embedding layer of the current training model to obtain the text vector. 9.A training device of a multi-label classification model, characterized in that, comprise a memory and a processor; The memory stores a computer program, and the computer program is executed by the processor to implement the steps of the multi-label classification model training method according to any one of claims 1 to 5.

10. A storage medium, characterized by The memory stores a computer program, and the computer program is executed by the processor to implement the steps of the multi-label classification model training method according to any one of claims 1 to 5.

Images