Model training method and device, text classification method and device, equipment and medium
Through the two-time comparative learning training method, the problem of low text classification accuracy caused by noise label interference is solved, the accuracy of the text classification model is improved, and the efficient training of the model in the noisy label environment is ensured.
Patent Information
- Application Number
- CN202411132505.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
In the field of text classification in natural language processing, noise label interference seriously affects the generalization performance of deep learning models, resulting in inaccurate classification prediction. How to improve text classification accuracy has become an urgent problem.
Two-time comparison learning training method are used. During the first comparison learning training, the noise label is used as the positive label to build the corresponding negative label; during the second comparison learning training, the preset label is used to determine the positive and negative label pairs to reduce the positive label. The distance from the training data is increased, the distance between the negative label and the training data is avoided, and the accuracy of the classification model is improved.
Through two comparative learning training, the classification accuracy of the text classification model is effectively improved, the negative impact of noise labels on model training is avoided, and the accuracy of prediction is improved.
Smart Images

Figure CN119089198A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method, text classification method, device, equipment and medium. Background Art
[0002] In recent years, with the deepening of deep learning research, related studies have found that the achievements of deep neural networks in many scenarios are inseparable from a large amount of real label information, and high-quality data is the basis for the training of its classification model. However, in real and complex life scenarios, noisy labels are a normal phenomenon, and learning from noisy labels is also a huge challenge. In the field of text classification in natural language processing, there is also the problem of noisy label interference, which will seriously affect the generalization performance of deep learning models and lead to inaccurate text classification predictions. Therefore, in the presence of noisy label interference, how to improve the classification accuracy of text has become an urgent problem to be solved. Summary of the invention
[0003] In view of this, embodiments of the present invention provide a model training method, a text classification method, an apparatus, a device and a medium to solve the problem of low text classification accuracy in the presence of noise label interference.
[0004] In a first aspect, an embodiment of the present invention provides a training method for a text classification model, the training method comprising:
[0005] Acquire training data with a label, use the label of the training data as a first positive label, and construct a first negative label for the training data according to the first positive label;
[0006] Obtain a pre-trained classification model, and perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label, and the first negative label to obtain a first trained classification model;
[0007] Obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, and determine the first K preset labels that best match the training data among the at least one preset label, where K is an integer greater than zero;
[0008] Using the preset labels among the K preset labels that are equal to the labels of the training data as the second positive labels, and using the remaining preset labels as the second negative labels;
[0009] According to the training data, the second positive label and the second negative label, the first trained classification model is subjected to a second comparative learning training to obtain a second trained classification model, and the second trained classification model is used as a trained classification model.
[0010] In a second aspect, an embodiment of the present invention provides a text classification method, the text classification method comprising:
[0011] Get the text to be classified;
[0012] The text to be classified is input into a trained classification model, and features are extracted from the text to be classified to obtain a feature vector of the text to be classified. Based on the label vector of each preset label and the feature vector, the similarity between the feature vector and each label vector is calculated. Based on the similarity, a target preset label matching the text to be classified is output, and the target preset label is used as the classification result of the text to be classified.
[0013] In a third aspect, an embodiment of the present invention provides a training device for a text classification model, the training device comprising:
[0014] A first construction module is used to obtain training data with a label, use the label of the training data as a first positive label, and construct a first negative label for the training data according to the first positive label;
[0015] A first training module is used to obtain a pre-trained classification model, and perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label and the first negative label to obtain a first trained classification model;
[0016] A computing module, configured to obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, and determine, among the at least one preset label, first K preset labels that best match the training data, where K is an integer greater than zero;
[0017] A second construction module is used to use the preset labels equal to the labels of the training data among the K preset labels as second positive labels, and use the remaining preset labels as second negative labels;
[0018] The second training module is used to perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label to obtain a second trained classification model, and use the second trained classification model as the trained classification model.
[0019] In a fourth aspect, an embodiment of the present invention provides a text classification device, the text classification device comprising:
[0020] An acquisition module is used to obtain the text to be classified;
[0021] The output module is used to input the text to be classified into a trained classification model, perform feature extraction on the text to be classified, obtain a feature vector of the text to be classified, calculate the similarity between the feature vector and each label vector based on the label vector of each preset label and the feature vector, output a target preset label that matches the text to be classified based on the similarity, and use the target preset label as the classification result of the text to be classified.
[0022] In a fifth aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for the text classification model as described in the first aspect or the text classification method as described in the second aspect is implemented.
[0023] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the text classification model as described in the first aspect or the text classification method as described in the second aspect is implemented.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] Acquire labeled training data, use the label of the training data as a first positive label, construct a first negative label for the training data according to the first positive label, obtain a pre-trained classification model, perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label and the first negative label, obtain a first trained classification model, obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, determine the first K preset labels that best match the training data among the at least one preset label, K is an integer greater than zero, use the preset labels among the K preset labels that are equal to the labels of the training data as second positive labels, use the remaining preset labels as second negative labels, perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label, obtain a second trained classification model, and use the second trained classification model as the trained classification model. In the present application, when training a text classification model using training data with noisy labels, two comparative learning trainings are used. In the first comparative learning training, the noisy label is used as the positive label, and the corresponding negative label is constructed. The positive label and the training data constitute a positive sample pair, and the negative label and the training data constitute a negative sample pair. Comparative learning training is performed based on the positive sample pairs and the negative sample pairs to obtain the classification model after the first training. In the second comparative learning training, the corresponding preset labels are used, and the corresponding labels and negative labels are determined in the preset labels, and they respectively constitute corresponding positive sample pairs and negative sample pairs with the training data. The classification model after the first training is subjected to comparative learning training again, so that when the classification model is trained using the training data with noisy labels, the influence of the noisy labels is avoided, thereby improving the classification accuracy of the classification model. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0027] Figure 1 It is a schematic diagram of an application environment of a training method and a text classification method provided by an embodiment of the present invention;
[0028] Figure 2 It is a flowchart of a training method for a text classification model provided by an embodiment of the present invention;
[0029] Figure 3 It is a flowchart of a text classification method provided by an embodiment of the present invention;
[0030] Figure 4It is a structural schematic diagram of a training device for a text classification model provided by an embodiment of the present invention;
[0031] Figure 5 is a structural schematic diagram of a text classification device provided by an embodiment of the present invention;
[0032] Figure 6 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.
[0035] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0036] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0037] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.
[0038] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0039] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0040] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0041] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0042] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0043] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.
[0044] Figure 1 1 is a schematic diagram of an application environment of a training method and a text classification method provided by an embodiment of the present invention. A training method of a text classification model provided by an embodiment of the present invention can be applied in Figure 1In an application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0045] See also Figure 2 , is a flowchart of a text classification model training method provided by an embodiment of the present invention. The text classification model training method can be applied to Figure 1 The server in Figure 2 As shown, the training method of the text classification model may include the following steps.
[0046] S201: Acquire labeled training data, use the label of the training data as a first positive label, and construct a first negative label for the training data according to the first positive label.
[0047] In step S201, labeled training data is obtained, wherein the label of the training data is a noisy label, that is, the label of the training data may be correct or incorrect, and the label of the training data is used as a first positive label, and a first negative label is constructed for the training data according to the first positive label, that is, other labels different from the first positive label are constructed.
[0048] In this embodiment, training data with labels is obtained, and the training data is text. The label of the training data is used as the first positive label, that is, the positive sample corresponding to the training data, which can form a corresponding positive sample pair with the training data, and the first negative label is constructed for the training data according to the first positive label. The first negative label is a negative sample corresponding to the training data, and when constructing the first negative label, other labels different from the first positive label are constructed.
[0049] S202: Obtain a pre-trained classification model, and perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label, and the first negative label to obtain a first trained classification model.
[0050] In step S202, a pre-trained classification model is obtained, wherein the pre-trained classification model is a model obtained after training a classification model based on the prior art, and a first comparative learning training is performed on the pre-trained classification model according to the training data, the first positive label, and the first negative label. During the comparative learning training, the training data and the first positive label are used as positive sample pairs, and the training data and the first negative label are used as negative sample pairs. The positive sample pairs and the negative samples are used to perform a first comparative learning training on the pre-trained classification model to obtain a first trained classification model.
[0051] In this embodiment, a pre-trained classification model is obtained, and a first comparative learning training is performed on the pre-trained classification model based on the training data, the first positive label and the first negative label. When performing the comparative learning training, the first positive label and the first negative label are used as positive samples and negative samples of the training data, so that when the labels are used as a recall library, labels matching the data to be classified can be retrieved.
[0052] It should be noted that when performing the first comparative learning training on the pre-trained classification model, the training can be stopped when the loss value is less than a preset threshold, or when the loss value converges, or when the number of training times reaches a preset number.
[0053] In this embodiment, the pre-trained classification model is subjected to the first comparative learning training based on the training data, the first positive label and the first negative label to reduce the distance between the first positive label and the training data and increase the distance between the first negative label and the training data, so that the pre-trained classification model learns the first positive label of the training data, and improves the ability to retrieve the correct label when the classification model is used to retrieve the corresponding label.
[0054] Optionally, performing a first comparative learning training on the pre-trained classification model according to the training data, the first positive label, and the first negative label to obtain a first trained classification model includes:
[0055] Input the training data, the first positive label, and the first negative label into a pre-trained classification model, perform feature extraction on the training data, the first positive label, and the first negative label to obtain a first feature vector of the training data, a first positive label vector of the first positive label, and a first negative label vector of the first negative label;
[0056] Calculating a first contrastive learning loss based on the first feature vector, the first positive label vector, and the first negative label vector;
[0057] According to the first contrastive learning loss, the pre-trained classification model is trained for the first contrastive learning to obtain a classification model after the first training.
[0058] In this embodiment, the training data, the first positive label, and the first negative label are input into the pre-trained classification model, and feature extraction is performed on the training data, the first positive label, and the first negative label to obtain a first feature vector of the training data, a first positive label vector of the first positive label, and a first negative label vector of the first negative label. According to the first feature vector, the first positive label vector, and the first negative label vector, a first contrastive learning loss is calculated, wherein the calculation formula of the first contrastive learning loss is as follows:
[0059]
[0060] Among them, L NCE1 is the first contrastive learning loss, is the first contrastive learning loss function, x is the corresponding training data, x + is the first positive label, is the i-th first negative label, N is the number of first negative labels, f(x) is the first eigenvector of the training data, f(x + ) is the first positive label vector of the first positive label, f(x - ) is the first negative label vector of the first negative label, is the distance between the first eigenvector and the first positive label vector, that is, the distance between the training data and the first positive label, is the distance between the first feature vector and the first negative label vector, that is, the distance between the training data and the first negative label.
[0061] According to the first contrastive learning loss, the pre-trained classification model is subjected to the first contrastive learning training to obtain the classification model after the first training, wherein the smaller the first contrastive learning loss is, the smaller the distance between the training data and the first positive label is, and the larger the distance between the training data and the first negative label is, that is, the training data is closer to the first positive label. Therefore, when the pre-trained classification model is subjected to the first contrastive learning training, the training can be stopped when the first contrastive loss is less than a preset threshold.
[0062] S203: Obtain at least one preset tag, retrieve and classify the training data according to the first trained classification model, and determine the top K preset tags that best match the training data from the at least one preset tag.
[0063] In step S203, at least one preset label is obtained, wherein the preset label includes a correct label corresponding to the training data, and is used to retrieve a label that matches the training data. The training data is retrieved and classified according to the first trained classification model, and among the at least one preset label, the top K preset labels that best match the training data are determined, where K is an integer greater than zero, such as K is 10.
[0064] In this embodiment, the first trained classification model is used to perform classification retrieval on the training data to determine the top K preset labels that best match the training data, wherein the top K preset labels include the correct labels of the training data.
[0065] In this embodiment, when determining the top K preset labels that best match the training data, the classification results of the training data can be compared with each preset label based on the classification model after the first training to determine the similarity between the classification results and each preset label, and the top K preset labels that best match the training data can be determined based on the similarity.
[0066] Optionally, the training data is retrieved and classified according to the first trained classification model, and top K preset labels that best match the training data are determined from at least one preset label, including:
[0067] Input the training data and each preset label into the first trained classification model, perform feature extraction on each preset label and the training data, and obtain a label vector of each preset label and a first feature vector of the training data;
[0068] The similarity between the first feature vector and the label vector of each preset label is calculated, and based on the similarity, the first K preset labels that best match the training data are output.
[0069] In this embodiment, the training data and each preset label are input into the first trained classification model, and feature extraction is performed on each preset label and the training data to obtain the label vector of each preset label and the first feature vector of the training data, and the similarity between the first feature vector and the label vector of each preset label is calculated. According to the similarity between the preset label and the training data, the top K preset labels that are most similar to the training data are calculated, wherein the top K preset labels include the correct labels of the training data, so that the positive samples corresponding to the training data and the corresponding negative samples can be determined from the top K preset labels to form positive sample pairs and negative sample pairs, and the positive sample pairs and negative sample pairs are used to perform comparative learning training on the first trained classification model.
[0070] S204: Using the preset labels that are equal to the labels of the training data among the K preset labels as the second positive labels, and using the remaining preset labels as the second negative labels.
[0071] In step S204, the preset labels among the K preset labels that are equal to the labels of the training data are used as the second positive labels, that is, the positive samples corresponding to the training data, and the remaining preset labels are used as the second negative labels, that is, the negative samples of the training data, where the number of negative samples can be multiple.
[0072] In this embodiment, the preset labels among the K preset labels that are equal to the labels of the training data are used as the second positive labels, which can constitute corresponding positive sample pairs with the training data, and the remaining preset labels are used as the second negative labels, which can constitute corresponding negative sample pairs with the training data, wherein there can be multiple negative sample pairs.
[0073] In this embodiment, corresponding positive labels and negative labels are determined according to K preset labels, so that they can be used as corresponding positive samples and negative samples when performing comparative learning training on the classification model after the first training.
[0074] S205: Perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label to obtain a second trained classification model, and use the second trained classification model as the trained classification model.
[0075] In step S205, the training data, the second positive label and the second negative label are used to perform a second comparative learning training on the first trained classification model. During the comparative learning training, the training data and the second positive label are used as positive sample pairs, and the training data and the second negative label are used as negative sample pairs. The positive sample pairs and the negative samples are used to perform a second comparative learning training on the first trained classification model to obtain a second trained classification model.
[0076] In this embodiment, a second comparative learning training is performed on the classification model after the first training based on the training data, the second positive label and the second negative label. When performing the comparative learning training, the second positive label and the second negative label are used as positive samples and negative samples of the training data, so that when the label is used as a recall library, the label matching the data to be classified can be retrieved.
[0077] In this embodiment, a second comparative learning training is performed on the classification model after the first training based on the training data, the second positive label and the second negative label to reduce the distance between the second positive label and the training data and increase the distance between the second negative label and the training data, so that the pre-trained classification model learns the second positive label of the training data, and improves the ability to retrieve the correct label when using the classification model to retrieve the corresponding label.
[0078] Optionally, performing a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label to obtain the second trained classification model includes:
[0079] Input the training data, the second positive label, and the second negative label into the pre-trained classification model, perform feature extraction on the training data, the second positive label, and the second negative label to obtain a second feature vector of the training data, a second positive label vector of the second positive label, and a second negative label vector of the second negative label;
[0080] Calculating a second contrastive learning loss according to the second feature vector, the second positive label vector, and the second negative label vector;
[0081] According to the second contrastive learning loss, a second contrastive learning training is performed on the first trained classification model to obtain a second trained classification model.
[0082] In this embodiment, the training data, the second positive label, and the second negative label are input into the pre-trained classification model, and feature extraction is performed on the training data, the second positive label, and the second negative label to obtain a second feature vector of the training data, a second positive label vector of the second positive label, and a second negative label vector of the second negative label. According to the second feature vector, the second positive label vector, and the second negative label vector, the second contrastive learning loss is calculated, wherein the calculation formula of the second contrastive loss is as follows:
[0083]
[0084] Among them, L NCE2 is the second contrast loss, is the second contrast loss function, y is the corresponding training data, y + is the second positive label, is the i-th second negative label, K-1 is the number of second negative labels, f(y) is the second eigenvector of the training data, f(y + ) is the second positive label vector, f(y - ) is the second negative label vector, is the distance between the second eigenvector and the second positive label vector, that is, the distance between the training data and the second positive label, is the distance between the second feature vector and the second negative label vector, that is, the distance between the training data and the second negative label.
[0085] It should be noted that the number of second negative labels can be changed, and some other second negative labels that are semantically different from the second positive labels can be added to make the number of second negative labels greater than K-1.
[0086] According to the second contrastive learning loss, the classification model after the first training is subjected to a second contrastive learning training to obtain the classification model after the second training. The smaller the second contrastive learning loss is, the smaller the distance between the training data and the second positive label is, and the larger the distance between the training data and the second negative label is, that is, the training data is closer to the second positive label. Therefore, when the classification model after the first training is subjected to a second contrastive learning training, the training can be stopped when the second contrastive loss is less than a preset threshold.
[0087] Optionally, after calculating the second contrastive learning loss according to the second feature vector, the second positive label vector, and the second negative label vector, the method further includes:
[0088] Calculate the penalty loss based on the second feature vector, the second positive label vector and the second negative label vector;
[0089] Calculate the second loss based on the second contrastive learning loss and the penalty loss;
[0090] According to the second loss, the first trained classification model is trained for the second time using contrastive learning to obtain the second trained classification model.
[0091] In this embodiment, in order to improve the classification accuracy of the classification model, it is avoided that when the second positive label is an incorrect label, the first trained classification model learns the wrong features, resulting in a low classification accuracy of the trained classification model. Therefore, the second contrastive learning loss is corrected, that is, a penalty term is added on the basis of the second contrastive learning loss as the total loss, that is, the second loss, and the second contrastive learning training is performed on the first trained classification model using the second loss to improve the classification accuracy of the trained classification model.
[0092] In this embodiment, the penalty loss is calculated according to the second eigenvector, the second positive label vector and the second negative label vector, wherein the penalty loss is calculated as follows:
[0093]
[0094] Among them, L λ is the penalty loss, n is the number of training data, λ is a hyperparameter related to the number of iterations and the number of training data, and p j is the probability distribution vector of the jth training data, t j is the iteration vector, p j ,t j The vector dimension of is K, where K is the number of the second positive label and the second negative label. <p j ,t j > for p j With t j The inner product of j The calculation formula is as follows:
[0095] t j (k) = βt j (k-1)+(1-β)p j (k)
[0096] Among them, t j (k) is the iteration vector at the kth iteration, t j (k-1) is the iteration vector for the k-1th iteration, p j (k) is the probability distribution of training data at the kth iteration, and β is a hyperparameter.
[0097] The calculation formulas of λ and β are as follows:
[0098]
[0099]
[0100] Among them, λ max is the maximum value of λ, such as 10, etc., β max The maximum value of β is set, such as 1, k is the kth iteration, and epochs is the number of times the model performs a complete traversal of the training data set.
[0101] According to the second contrastive learning loss and the penalty loss, the second loss is calculated. The calculation formula of the second loss is as follows:
[0102] L=L NCE2 +L λ
[0103] Among them, L is the second loss, L NCE2 is the second contrast loss, L λ To punish losses.
[0104] In this embodiment, a penalty term is added on the basis of the second contrastive learning loss as the total loss, i.e., the second loss, so that the model is corrected within K labels, and the second contrastive learning training is performed on the first trained classification model according to the second loss to obtain the second trained classification model. The robustness of the second trained classification model is increased.
[0105] Acquire labeled training data, use the label of the training data as a first positive label, construct a first negative label for the training data according to the first positive label, obtain a pre-trained classification model, perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label and the first negative label, obtain a first trained classification model, obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, determine the first K preset labels that best match the training data among the at least one preset label, K is an integer greater than zero, use the preset labels among the K preset labels that are equal to the labels of the training data as second positive labels, use the remaining preset labels as second negative labels, perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label, obtain a second trained classification model, and use the second trained classification model as the trained classification model. In the present application, when training a text classification model using training data with noisy labels, two comparative learning trainings are used. In the first comparative learning training, the noisy label is used as the positive label, and the corresponding negative label is constructed. The positive label and the training data constitute a positive sample pair, and the negative label and the training data constitute a negative sample pair. Comparative learning training is performed based on the positive sample pairs and the negative sample pairs to obtain the classification model after the first training. In the second comparative learning training, the corresponding preset labels are used, and the corresponding labels and negative labels are determined in the preset labels, and they respectively constitute corresponding positive sample pairs and negative sample pairs with the training data. The classification model after the first training is subjected to comparative learning training again, so that when the classification model is trained using the training data with noisy labels, the influence of the noisy labels is avoided, thereby improving the classification accuracy of the classification model.
[0106] See also Figure 3 , is a flowchart of a text classification method provided by an embodiment of the present invention, such as Figure 3 As shown, the text classification method may include the following steps:
[0107] S301: Obtaining the text to be classified;
[0108] S302: Input the text to be classified into the trained classification model, perform feature extraction on the text to be classified, obtain the feature vector of the text to be classified, calculate the similarity between the feature vector and each label vector based on the label vector and feature vector of each preset label, output the target preset label that matches the text to be classified based on the similarity, and use the target preset label as the classification result of the text to be classified.
[0109] In this embodiment, the training steps of the trained classification model include:
[0110] Acquire labeled training data, use the label of the training data as a first positive label, construct a first negative label for the training data according to the first positive label, obtain a pre-trained classification model, perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label and the first negative label, obtain a first trained classification model, obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, determine the first K preset labels that best match the training data among the at least one preset label, K is an integer greater than zero, use the preset labels among the K preset labels that are equal to the labels of the training data as second positive labels, use the remaining preset labels as second negative labels, perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label, obtain a second trained classification model, and use the second trained classification model as the trained classification model.
[0111] According to the training data, the first positive label and the first negative label, the pre-trained classification model is trained for the first time by contrastive learning to obtain a classification model after the first training, including:
[0112] The training data, the first positive label, and the first negative label are input into the pre-trained classification model, and features are extracted from the training data, the first positive label, and the first negative label to obtain a first feature vector of the training data, a first positive label vector of the first positive label, and a first negative label vector of the first negative label; a first contrastive learning loss is calculated based on the first feature vector, the first positive label vector, and the first negative label vector; based on the first contrastive learning loss, the pre-trained classification model is trained for the first contrastive learning to obtain a first trained classification model.
[0113] According to the first trained classification model, the training data is retrieved and classified, and among at least one preset label, the first K preset labels that best match the training data are determined, including:
[0114] The training data and each preset label are input into the classification model after the first training, and features are extracted for each preset label and the training data to obtain the label vector of each preset label and the first feature vector of the training data, and the similarity between the first feature vector and the label vector of each preset label is calculated. Based on the similarity, the top K preset labels that best match the training data are output.
[0115] According to the training data, the second positive label and the second negative label, the first trained classification model is subjected to a second comparative learning training to obtain the second trained classification model, including:
[0116] The training data, the second positive label, and the second negative label are input into the pre-trained classification model, and features are extracted from the training data, the second positive label, and the second negative label to obtain a second feature vector of the training data, a second positive label vector of the second positive label, and a second negative label vector of the second negative label. A second contrastive learning loss is calculated based on the second feature vector, the second positive label vector, and the second negative label vector. Based on the second contrastive learning loss, a second contrastive learning training is performed on the first trained classification model to obtain a second trained classification model.
[0117] After calculating the second contrastive learning loss according to the second feature vector, the second positive label vector and the second negative label vector, the further step further includes:
[0118] The penalty loss is calculated according to the second eigenvector, the second positive label vector and the second negative label vector. The second loss is calculated according to the second contrastive learning loss and the penalty loss. According to the second loss, the first trained classification model is subjected to a second contrastive learning training to obtain the second trained classification model.
[0119] In this embodiment, a text to be classified is obtained, wherein the label corresponding to the text to be classified is one of the preset labels, the text to be classified is input into a trained classification model, feature extraction is performed on the text to be classified, and a feature vector of the text to be classified is obtained, and the similarity between the feature vector and each label vector is calculated based on the label vector and the feature vector of each preset label, and based on the similarity, the preset label corresponding to the maximum similarity is output as the target label, that is, the preset label with the maximum similarity is used as the classification result of the text to be classified.
[0120] See also Figure 4 , Figure 4 Schematic diagram of a text classification model training device provided by an embodiment of the present invention. The units included in this embodiment are used to execute Figure 2 For details, please refer to the steps in the corresponding embodiment. Figure 2 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 4 The training device 40 includes: a first construction module 41, a first training module 42, a calculation module 43, a second construction module 44, and a second training module 45.
[0121] The first construction module 41 is used to obtain training data with labels, use the labels of the training data as first positive labels, and construct first negative labels for the training data according to the first positive labels.
[0122] The first training module 42 is used to obtain a pre-trained classification model, and perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label and the first negative label to obtain a first trained classification model.
[0123] The calculation module 43 is used to obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, and determine the first K preset labels that best match the training data among the at least one preset label, where K is an integer greater than zero.
[0124] A second construction module 44 is used to use the preset labels that are equal to the labels of the training data among the K preset labels as second positive labels, and use the remaining preset labels as second negative labels;
[0125] The second training module 45 is used to perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label to obtain a second trained classification model, and use the second trained classification model as the trained classification model.
[0126] Optionally, the first training module 42 includes:
[0127] The first input unit is used to input the training data, the first positive label and the first negative label into the pre-trained classification model, perform feature extraction on the training data, the first positive label and the first negative label, and obtain a first feature vector of the training data, a first positive label vector of the first positive label and a first negative label vector of the first negative label.
[0128] The first calculation unit is used to calculate a first contrastive learning loss according to the first feature vector, the first positive label vector and the first negative label vector.
[0129] The first training unit is used to perform a first contrastive learning training on the pre-trained classification model according to a first contrastive learning loss to obtain a first trained classification model.
[0130] Optionally, the calculation module 43 includes:
[0131] The extraction unit is used to input the training data and each preset label into the first trained classification model, perform feature extraction on each preset label and the training data, and obtain a label vector of each preset label and a first feature vector of the training data.
[0132] The output unit is used to calculate the similarity between the first feature vector and the label vector of each preset label, and output the first K preset labels that best match the training data according to the similarity.
[0133] Optionally, the second training module 45 includes:
[0134] The second input unit is used to input the training data, the second positive label and the second negative label into the pre-trained classification model, perform feature extraction on the training data, the second positive label and the second negative label, and obtain a second feature vector of the training data, a second positive label vector of the second positive label and a second negative label vector of the second negative label.
[0135] The second calculation unit is used to calculate a second contrastive learning loss according to the second feature vector, the second positive label vector and the second negative label vector.
[0136] The second training unit is used to perform a second contrastive learning training on the first trained classification model according to the second contrastive learning loss to obtain the second trained classification model.
[0137] Optionally, the second training module 45 further includes:
[0138] The third calculation unit is used to calculate the penalty loss according to the second feature vector, the second positive label vector and the second negative label vector.
[0139] The third calculation unit is used to calculate the second loss according to the second contrastive learning loss and the penalty loss.
[0140] The third training unit is used to perform a second contrast learning training on the first trained classification model according to the second loss to obtain the second trained classification model.
[0141] See also Figure 5 , Figure 5 Schematic diagram of the structure of a text classification device provided by an embodiment of the present invention. The units included in this embodiment are used to perform Figure 3 For details, please refer to the steps in the corresponding embodiment. Figure 3 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 5 The text classification device 50 includes: an acquisition module 51 and an output module 52.
[0142] The acquisition module 51 is used to acquire the text to be classified.
[0143] The output module 52 is used to input the text to be classified into the trained classification model, perform feature extraction on the text to be classified, obtain the feature vector of the text to be classified, calculate the similarity between the feature vector and each label vector based on the label vector and the feature vector of each preset label, and output the target preset label that matches the text to be classified based on the similarity, and use the target preset label as the classification result of the text to be classified.
[0144] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules, units, and sub-units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0145] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 6 As shown, the computer device of this embodiment includes: at least one processor ( Figure 6 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in the training method of any of the above-mentioned models or the text classification method embodiments are implemented.
[0146] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 6 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components.
[0147] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0148] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments may also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory may also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.
[0149] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0150] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.
[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0153] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0155] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A training method for a text classification model, characterized in that: The training method comprises: Acquire training data with a label, use the label of the training data as a first positive label, and construct a first negative label for the training data according to the first positive label; Obtain a pre-trained classification model, and perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label, and the first negative label to obtain a first trained classification model; Obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, and determine the first K preset labels that best match the training data among the at least one preset label, where K is an integer greater than zero; Using the preset labels among the K preset labels that are equal to the labels of the training data as the second positive labels, and using the remaining preset labels as the second negative labels; According to the training data, the second positive label and the second negative label, the first trained classification model is subjected to a second comparative learning training to obtain a second trained classification model, and the second trained classification model is used as a trained classification model.
2. The training method according to claim 1, characterized in that: The step of performing a first comparative learning training on the pre-trained classification model according to the training data, the first positive label, and the first negative label to obtain a first trained classification model includes: Inputting the training data, the first positive label, and the first negative label into the pre-trained classification model, performing feature extraction on the training data, the first positive label, and the first negative label to obtain a first feature vector of the training data, a first positive label vector of the first positive label, and a first negative label vector of the first negative label; Calculating a first contrastive learning loss based on the first feature vector, the first positive label vector, and the first negative label vector; According to the first contrastive learning loss, a first contrastive learning training is performed on the pre-trained classification model to obtain a first trained classification model.
3. The training method according to claim 1, characterized in that: The retrieving and classifying the training data according to the first trained classification model, and determining the first K preset labels that best match the training data from the at least one preset label, comprises: Inputting the training data and each preset label into the first trained classification model, performing feature extraction on each preset label and the training data, and obtaining a label vector of each preset label and a first feature vector of the training data; The similarity between the first feature vector and the label vector of each preset label is calculated, and based on the similarity, the first K preset labels that best match the training data are output.
4. The training method according to claim 1, characterized in that: The step of performing a second comparative learning training on the first trained classification model according to the training data, the second positive label, and the second negative label to obtain a second trained classification model includes: Inputting the training data, the second positive label, and the second negative label into the pre-trained classification model, performing feature extraction on the training data, the second positive label, and the second negative label to obtain a second feature vector of the training data, a second positive label vector of the second positive label, and a second negative label vector of the second negative label; Calculating a second contrastive learning loss based on the second feature vector, the second positive label vector, and the second negative label vector; According to the second contrastive learning loss, a second contrastive learning training is performed on the first trained classification model to obtain a second trained classification model.
5. The training method according to claim 4, characterized in that: After calculating the second contrastive learning loss according to the second feature vector, the second positive label vector and the second negative label vector, the method further includes: Calculate a penalty loss based on the second feature vector, the second positive label vector, and the second negative label vector; Calculating a second loss according to the second contrastive learning loss and the penalty loss; According to the second loss, a second contrastive learning training is performed on the first trained classification model to obtain a second trained classification model.
6. A text classification method, characterized in that: The text classification method comprises: Get the text to be classified; The text to be classified is input into a classification model trained using the training method of claims 1-5, and features of the text to be classified are extracted to obtain a feature vector of the text to be classified. Based on the label vector of each preset label and the feature vector, the similarity between the feature vector and each label vector is calculated. Based on the similarity, a target preset label matching the text to be classified is output, and the target preset label is used as the classification result of the text to be classified.
7. A training device for a text classification model, characterized in that: The training device comprises: A first construction module is used to obtain training data with a label, use the label of the training data as a first positive label, and construct a first negative label for the training data according to the first positive label; A first training module is used to obtain a pre-trained classification model, and perform a first comparative learning training on the pre-trained classification model according to the training data, the first positive label and the first negative label to obtain a first trained classification model; A computing module, configured to obtain at least one preset label, retrieve and classify the training data according to the first trained classification model, and determine, among the at least one preset label, first K preset labels that best match the training data, where K is an integer greater than zero; A second construction module is used to use the preset labels equal to the labels of the training data among the K preset labels as second positive labels, and use the remaining preset labels as second negative labels; The second training module is used to perform a second comparative learning training on the first trained classification model according to the training data, the second positive label and the second negative label to obtain a second trained classification model, and use the second trained classification model as the trained classification model.
8. A text classification device, characterized in that: The text classification device comprises: An acquisition module is used to obtain the text to be classified; The output module is used to input the text to be classified into a trained classification model, perform feature extraction on the text to be classified, obtain a feature vector of the text to be classified, calculate the similarity between the feature vector and each label vector based on the label vector of each preset label and the feature vector, output a target preset label that matches the text to be classified based on the similarity, and use the target preset label as the classification result of the text to be classified.
9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the training method for the text classification model according to any one of claims 1 to 5 or the text classification method according to claim 6 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the training method of the text classification model according to any one of claims 1 to 5 or the text classification method according to claim 6 is implemented.
Citation Information
Patent Citations
Scatter diagram similarity measurement method, terminal equipment and storage medium
CN115564974A
Deep learning-based multi-label image identification method under noisy data
CN116012569A
Classification model training method and device, classification method and device, equipment and medium
CN116152551A
Pre-training classification model training method and device, equipment and medium
CN116522222A
Text classification model training method and device and text classification method and device
CN117076672A
