Hierarchical text classification method, model training method, system, equipment and medium
Through the dual prompt template and prompt label comparison learning algorithm, the hierarchical perception ability of the hierarchical text classification model is extracted, and the problem of similar characteristics of low-level hierarchical labels is solved, and classification accuracy and understanding of hierarchical structure are improved.
Patent Information
- Application Number
- CN202411067005.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-06
AI Technical Summary
In the hierarchical text classification task, the labels are granular in the low-level hierarchy, and the characteristics of the same parent node labels are similar, making it difficult to distinguish, resulting in low classification accuracy and it is difficult for the existing technology to effectively capture complex hierarchical structures and semantic relationships.
A hierarchical text classification method is proposed, which forms the training data through dual prompt templates, executes the prompt tag comparison learning algorithm, extracts the tag feature embedding, and calculates the comparison loss in combination with positive and negative tags, and trains the model to identify the correct tags and confusion. Negative labels, using cross-layer sorting losses and auxiliary task losses to improve the model's hierarchical perception capabilities.
It improves the accuracy of the hierarchical text classification task, can identify semantic features with strong discrimination and rich hierarchical information, alleviates the confusion problem between labels, and enhances the model's ability to understand hierarchical structures.
Smart Images

Figure CN119088961A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a hierarchical text classification method, model training method, system, device and medium. Background Art
[0002] Hierarchical Text Classification (HTC) is a special text classification task that aims to systematically divide text data into multiple categories with a hierarchical structure. It is suitable for scenarios where the categories form a certain system. Unlike traditional text classification, hierarchical text classification provides more refined and structured labels, making the classification results more flexible and interpretable. Hierarchical text classification technology has a wide range of applications in the industry. It can adapt to complex category systems, learn the relationships between different categories, and support more fine-grained classification. It is a key technology for the practical application of natural language processing.
[0003] The category system of hierarchical text classification is constructed and stored in a predefined hierarchical structure. There is a natural hierarchical constraint relationship between labels, such as the "parent-child" relationship between upper and lower level categories, and the "brother" relationship between the same level and the same parent node. The category system of hierarchical text classification has complex hierarchical structure and semantic relationships. For example, a parent category may be associated with multiple subcategories, and there may be some overlapping semantics between categories at the same level. Especially at the lower level, the label granularity is finer, and the features of sublabels on the same parent node label are relatively similar and difficult to distinguish, resulting in low accuracy of hierarchical text classification tasks. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a hierarchical text classification method, model training method, system, device and medium, aiming to improve the accuracy of hierarchical text classification tasks.
[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a hierarchical text classification model training method, comprising the following steps:
[0006] Obtain a training dataset annotated according to a predefined labeling architecture;
[0007] The sample data in the training data set is converted into a form according to a dual prompt template to obtain a prompt text, wherein the dual prompt template includes text data, a positive hierarchical relationship string and a negative hierarchical relationship string, the positive hierarchical relationship string includes a plurality of positive label mask characters representing the hierarchical relationship, and the negative hierarchical relationship string includes a plurality of negative label mask characters representing the hierarchical relationship;
[0008] Inputting the text data in the prompt text into the initialized hierarchical text classification model for forward transmission, obtaining positive label mask word vectors on positive label mask characters, negative label mask word vectors on negative label mask characters and prediction results, wherein the prediction results include hierarchical label categories and confidence scores of hierarchical label categories;
[0009] According to the positive label mask word vector, the true label vector and the negative label vector of the text data, positive label contrast learning is performed on the positive label mask character to obtain the positive label contrast loss;
[0010] According to the negative label mask word vector, the true label vector and the negative label vector of the text data, negative label contrast learning is performed on the negative label mask character to obtain a negative label contrast loss;
[0011] Determine the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result;
[0012] The parameters of the hierarchical text classification model are updated according to the classification task loss to obtain a trained hierarchical text classification model.
[0013] In some embodiments, determining the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result comprises the following steps:
[0014] Determine a positive sub-label word vector and a negative sub-label word vector of a target positive label mask word vector according to a hierarchical relationship of the positive label mask word vectors on a plurality of positive label mask characters;
[0015] Determine a cross-layer ranking loss according to a first similarity between the target positive label mask word vector and the positive sub-label word vector, and a second similarity between the target positive label mask word vector and the negative sub-label word vector;
[0016] Performing a weighted combination of the cross-layer ranking loss, the positive label contrast loss, and the negative label contrast loss to obtain a prompt label contrast learning loss;
[0017] The classification task loss is determined based on the prompt label comparison learning loss and the prediction result.
[0018] In some embodiments, the hierarchical text classification model training method further includes the following steps:
[0019] Performing identification word mapping on the sample label text to obtain an identification representation label of the sample label, wherein the sample label includes a true label text and a negative label text of the sample data;
[0020] Using a preset hierarchical structure representation form to perform hierarchical flattening representation processing on the identification representation labels with hierarchical structure information to obtain a label sequence;
[0021] The label sequence is input into the feature representation layer of the hierarchical text classification model to obtain a sample label vector, wherein the sample label vector includes a true label vector and a negative label vector.
[0022] In some embodiments, the negative label text is obtained by the following steps, including:
[0023] Sorting the hierarchical label categories output by the hierarchical text classification model according to the confidence scores from large to small;
[0024] The top preset number of hierarchical label categories are used as negative label texts.
[0025] In some embodiments, determining the classification task loss according to the prompt label comparison learning loss and the prediction result comprises the following steps:
[0026] Based on the cross entropy loss function, the consistency loss is determined according to the hierarchical relationship prediction score of the hierarchical label category, wherein the hierarchical relationship prediction score represents the prediction score of the hierarchical text classification model on whether the hierarchical label category conforms to the true hierarchical relationship;
[0027] Based on the cross entropy loss function, determining the classification correctness loss according to the semantic classification prediction score of the hierarchical label category, wherein the semantic classification prediction score represents the prediction score of the hierarchical text classification model on whether the hierarchical label category conforms to the true label;
[0028] Determining an auxiliary task loss according to the consistency loss and the classification correctness loss;
[0029] The classification task loss is determined according to the prompt label contrast learning loss, the auxiliary task loss, the model overall classification loss and the masked language model loss.
[0030] To achieve the above purpose, another aspect of the embodiment of the present application proposes a hierarchical text classification method, comprising the following steps:
[0031] Get the text to be inferred;
[0032] Performing form conversion on the text to be inferred according to the dual prompt template to obtain an input text;
[0033] The input text is input into a hierarchical text classification model to obtain a hierarchical text prediction result, and a hierarchical classification result of the text to be inferred is determined according to the hierarchical text prediction result, wherein the hierarchical text classification model is obtained by the hierarchical text classification model training method of the previous embodiment.
[0034] In some embodiments, the hierarchical text prediction result includes multiple hierarchical label categories and confidence scores of each hierarchical label category, and determining the hierarchical classification result of the text to be inferred according to the hierarchical text prediction result includes the following steps:
[0035] Determining whether the highest confidence score in the hierarchical text prediction result is greater than a preset score;
[0036] When the highest confidence score in the hierarchical text prediction result is greater than a preset score, the hierarchical classification result of the text to be inferred is determined according to the hierarchical text prediction result;
[0037] When the highest confidence score in the hierarchical text prediction result is less than or equal to a preset score, the hierarchical label categories with the second-preset number of confidence scores in the hierarchical text prediction result are taken as candidate hierarchical labels;
[0038] Filling the context learning prompt template of the large language model according to the candidate level labels to obtain a text classification question text;
[0039] The text classification question text is input into a large language model to obtain a text classification answer, and the text classification answer is analyzed and extracted to obtain a hierarchical classification result of the text to be inferred.
[0040] To achieve the above purpose, another aspect of the embodiment of the present application proposes a hierarchical text classification model training system, including:
[0041] The first module is used to obtain a training dataset annotated according to a predefined label architecture;
[0042] The second module is used to perform form conversion on the sample data in the training data set according to the dual prompt template to obtain a prompt text, wherein the dual prompt template includes text data, a positive hierarchical relationship string and a negative hierarchical relationship string, the positive hierarchical relationship string includes a plurality of positive label mask characters representing the hierarchical relationship, and the negative hierarchical relationship string includes a plurality of negative label mask characters representing the hierarchical relationship;
[0043] The third module is used to input the text data in the prompt text into the initialized hierarchical text classification model for forward transmission, and obtain the positive label mask word vector on the positive label mask character, the negative label mask word vector on the negative label mask character and the prediction result, wherein the prediction result includes the hierarchical label category and the confidence score of the hierarchical label category;
[0044] The fourth module is used to perform positive label contrast learning on the positive label mask characters according to the positive label mask word vector, the true label vector and the negative label vector of the text data to obtain the positive label contrast loss; perform negative label contrast learning on the negative label mask characters according to the negative label mask word vector, the true label vector and the negative label vector of the text data to obtain the negative label contrast loss; determine the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result;
[0045] The fifth module is used to update the parameters of the hierarchical text classification model according to the classification task loss to obtain a trained hierarchical text classification model.
[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the hierarchical text classification model training method or the hierarchical text classification method described in the above-mentioned embodiment is realized.
[0047] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the hierarchical text classification model training method or hierarchical text classification method described in the above-mentioned embodiment.
[0048] The hierarchical text classification method, model training method, system, device and medium proposed in the present application obtain prompt text by converting the form of sample data in the training data set according to the dual prompt template, and executes the hierarchical-aware prompt label contrast learning algorithm based on the dual prompt template, that is, extracting the feature embedding of the label prompt word, and then mapping the embedding on the label mask bit to the category name, and further combining the true label and the negative label to calculate the positive label contrast loss and the negative label contrast loss. The hierarchical text classification model is trained by using the classification task loss that combines the positive label contrast loss and the negative label contrast loss, so that the model can not only identify the correct label, but also identify the erroneous negative label that tends to be confused with the positive label, which helps to extract semantic features with strong discriminative power and rich in hierarchical information, thereby improving the accuracy of the hierarchical text classification task. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a flowchart of a hierarchical text classification model training method provided in an embodiment of the present application;
[0050] Figure 2 yes Figure 1 Flow chart of step S106 in FIG.
[0051] Figure 3 is a flowchart of a hierarchical text classification model training method provided by another embodiment of the present application;
[0052] Figure 4 yes Figure 3 Flow chart of the method for obtaining negative label text in step S301 in ;
[0053] Figure 5 yes Figure 2 Flow chart of step S204 in FIG.
[0054] Figure 6 is a flowchart of a hierarchical text classification method provided in an embodiment of the present application;
[0055] Figure 7 yes Figure 6 Flow chart of step S603 in FIG.
[0056] Figure 8 is a schematic diagram of a hierarchical text classification model training system provided in an embodiment of the present application;
[0057] Fig. 9 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;
[0058] Fig.10 is a schematic diagram of a label system structure provided by an embodiment of the present application;
[0059] Fig.11It is a flowchart of the overall training process of the hierarchical text classification model provided in the embodiment of the present application;
[0060] Fig.12 It is a schematic diagram of the hierarchical text classification process combining a small model and a large language model provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0064] First, some nouns involved in this application are analyzed:
[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0066] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.
[0067] The Large Language Model (LLM) is trained based on massive text data and has powerful natural language processing (NLP) capabilities. It can perform various complex language tasks such as text generation, text classification, question-answering systems, etc.
[0068] In hierarchical text classification, label categories are organized into a directed acyclic graph with a hierarchical structure, usually in the form of a label tree, where each category can be regarded as a node, which is covered by the parent category and can be differentiated into subcategories. For example, Fig.10 This is an example of a label tree of a three-layer classification system for telephone traffic data in the telecommunications field. The goal of hierarchical text classification is to classify text or documents layer by layer starting from the root node according to the label tree structure, so as to obtain one or more fine-grained labels, such as "query-cost status-arrears query", corresponding to a path in the tree. Hierarchical text classification has applications in many fields in real scenarios, such as intent recognition in intelligent customer service systems and work order classification in government management platforms.
[0069] At present, the category system of hierarchical text classification has a complex hierarchy. The category system of hierarchical text classification is constructed and stored in a predefined hierarchical structure. There is a natural hierarchical constraint relationship between labels. How to learn and utilize the hierarchical information becomes the difficulty and key of the hierarchical text classification problem. Using graph neural network modeling is a commonly used method at present, but these methods are relatively simple and do not fully extract the complex semantic associations between text and labels, and between labels. Categories in hierarchical text classification usually have more complex semantic relationships. For example, a parent category may be associated with multiple subcategories, and there may be partially overlapping semantics between categories at the same level. Especially at the lower level, the label granularity is finer, and the features between sublabels on the same parent node label are relatively similar and difficult to distinguish, while the semantic encoding and discrimination capabilities of the models used in related technologies are insufficient.
[0070] In related technologies, a "dual encoder" architecture is usually used for text hierarchical label classification, that is, a text encoder and a structural encoder are used to model text data and hierarchical structures respectively. The text encoder uses a pre-trained language model, while the structural encoder most often uses a graph neural network. The two-way features are then fused to obtain text features that are aware of the label hierarchy. The enhanced text feature vector is then subjected to hierarchical label classification using a feedforward neural network. However, the "dual encoder" architecture introduces additional model parameters and simply fuses structural information directly into text features, making it difficult to capture highly discriminative semantic relationships. It only focuses on explicit superior-subordinate label associations in the hierarchy, and the interactions between labels at the same level are often ignored, making it easier for labels at the same level to be confused. The label hierarchy relationship during hierarchical text classification modeling is usually used to enhance text feature expression, and is rarely modeled as a separate learning task, resulting in the model not actually learning the label hierarchy relationship knowledge and being unable to accurately classify hierarchical text.
[0071] Based on this, the embodiments of the present application provide a hierarchical text classification method, model training method, system, device and medium, aiming to improve the accuracy of hierarchical text classification tasks.
[0072] The hierarchical text classification method, model training method, system, device and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the hierarchical text classification model training method and the hierarchical text classification method in the embodiments of the present application are described.
[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0075] The hierarchical text classification model training method and hierarchical text classification method provided in the embodiment of the present application relate to the field of artificial intelligence technology. The hierarchical text classification model training method or hierarchical text classification method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a hierarchical text classification model training method or a hierarchical text classification method, but is not limited to the above forms.
[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0077] Figure 1 is an optional flowchart of the hierarchical text classification model training method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.
[0078] Step S101, obtaining a training data set annotated according to a predefined label architecture;
[0079] Step S102, converting the sample data in the training data set into a form according to the dual prompt template to obtain a prompt text, wherein the dual prompt template includes text data, a positive hierarchical relationship string and a negative hierarchical relationship string, the positive hierarchical relationship string includes a plurality of positive label mask characters representing the hierarchical relationship, and the negative hierarchical relationship string includes a plurality of negative label mask characters representing the hierarchical relationship;
[0080] Step S103, inputting the text data in the prompt text into the initialized hierarchical text classification model for forward transmission, obtaining the positive label mask word vector on the positive label mask character, the negative label mask word vector on the negative label mask character and the prediction result, wherein the prediction result includes the hierarchical label category and the confidence score of the hierarchical label category;
[0081] Step S104, performing positive label contrast learning on the positive label mask character according to the positive label mask word vector, the true label vector and the negative label vector of the text data to obtain the positive label contrast loss;
[0082] Step S105, performing negative label contrast learning on the negative label mask character according to the negative label mask word vector, the true label vector and the negative label vector of the text data to obtain the negative label contrast loss;
[0083] Step S106, determining the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result;
[0084] Step S107, updating the parameters of the hierarchical text classification model according to the classification task loss to obtain a trained hierarchical text classification model.
[0085] Steps S101 to S107 shown in the embodiment of the present application are to obtain prompt text by converting the sample data in the training data set according to the dual prompt template, and to perform a hierarchical-aware prompt label contrast learning algorithm based on the dual prompt template, that is, to extract the feature embedding of the label prompt word, and then map the embedding on the label mask bit to the category name, and further calculate the positive label contrast loss and the negative label contrast loss in combination with the true label and the negative label. The hierarchical text classification model is trained using the classification task loss that combines the positive label contrast loss and the negative label contrast loss, so that the model can not only identify the correct label, but also the erroneous negative label that tends to be confused with the positive label, which helps to extract semantic features with strong discriminative power and rich in hierarchical information, thereby improving the accuracy of the hierarchical text classification task.
[0086] In step S101 of some embodiments, a domain expert may predefine a set of hierarchical label architectures, usually in the form of a label tree, according to business background and requirements. N} to obtain the training data set, where t i Represents the i-th text data.
[0087] In step S102 of some embodiments, a dual prompt template is constructed to perform form conversion on the sample data. Specifically, the dual prompt template retains two types of prompt word slots, positive label and negative label. The dual prompt template is as follows:
[0088] T=
[0089] {[CLS]t i [SEP]This text belongs to [MASK]1-…-[MASK] L , but not [MASK]1-…-[MASK] L [SEP];
[0090] Among them, [CLS], [SEP], and [MASK] are all special character identifiers in the language model. In this example, [MASK] is used as a "label mask character". [MASK]1-…-[MASK] L Indicates the first-level label to the last-level label. The first set of label mask characters (i.e., positive hierarchical relationship string) is used to indicate the predicted positive label, and the second set of label mask characters (i.e., negative hierarchical relationship string) is used to indicate the predicted negative label. L represents the number of levels of the label system.
[0091] According to the dual prompt template, the sample data in the training data set is converted into a new text form with prompt information to obtain the prompt text. Specifically, the text t in the training data set is converted into a new text form with prompt information. i Write the text data part of the dual prompt template, write the annotated true label into the first set of label mask characters in the template, and write the negative label into the second set of label mask characters.
[0092] In step S103 of some embodiments, the hierarchical text classification model may use the BERT language model as a backbone network for encoding. When a text sequence is input into the hierarchical text classification model, the text is first encoded by the BERT network to obtain a word vector for each character, which is expressed as follows:
[0093] V = BERT(T(t i ));
[0094] Text Data i After the BERT network, the positive label mask word vector on the positive label mask character and the negative label mask word vector on the negative label mask character can be extracted. This embodiment uses represents the positive label mask word vector of the lth layer, Represents the negative label mask word vector.
[0095] Based on the Multi-Verbalizers framework, each level The vector is mapped to one or more label words and a set of prediction results {(la1, sco1), (la2, sco2), ...} is output. j Indicates the predicted level label category, scoj Indicates the confidence score of the model on the level label category. Each level label category can represent the category node to which the text belongs and the label system path containing the level information.
[0096] In steps S104 to S105 of some embodiments, in order to capture discriminable semantic features containing hierarchical information, this embodiment proposes a hierarchical-aware prompt label contrast learning algorithm based on dual prompt fine-tuning. The algorithm encourages that the embedding vector of each label mask in the prompt template should be close to the representation of the label it should actually fill in. Its objective function includes three parts: positive label contrast learning, negative label contrast learning, and cross-layer ranking loss.
[0097] Positive label contrast learning is performed on positive label mask characters, with the goal of making the positive label mask word vector and the true label vector as close as possible. The positive samples are the M true labels of the text, and the negative samples are the K sampled negative labels and negative label mask characters. The positive label contrast loss formula is as follows:
[0098]
[0099] Negative label contrast learning is performed on negative label mask characters. Its purpose is to make the negative label mask word vector and the negative label vector as close as possible. The positive sample is the K sampled negative labels, and the negative sample is the M positive labels and positive label mask characters. The negative label contrast loss formula is as follows:
[0100]
[0101] in, They represent the mth positive label representation vector (i.e., true label vector) and the kth negative label representation vector (i.e., negative label vector) of the lth layer respectively. s(·) represents the cosine similarity function, and τ is the temperature coefficient parameter.
[0102] In step S106 of some embodiments, after the positive label contrast loss and the negative label contrast loss are calculated, the positive label contrast loss and the negative label contrast loss are integrated to determine the prompt label contrast learning loss, and then the classification task loss of the entire hierarchical text classification model is calculated in combination with the prompt label contrast learning loss.
[0103] In step S107 of some embodiments, the parameters of the hierarchical text classification model are reversely updated according to the classification task loss, and the hierarchical text classification model is continuously batch trained through the above process until a preset condition is met to obtain a trained hierarchical text classification model. The preset condition may be that a preset number of training times is reached or that the prediction result of the hierarchical text classification model reaches a preset accuracy. By calculating the positive label contrast loss and the negative label contrast loss on the positive label mask and the negative label mask, the prompt label contrast learning loss is obtained, and the model is updated in combination with the loss indication, so that the model can distinguish negative labels close to the true labels, and the accuracy of the hierarchical text classification is obtained.
[0104] See also Figure 2 In some embodiments, step S106 may include but is not limited to steps S201 to S204:
[0105] Step S201, determining a positive sub-label word vector and a negative sub-label word vector of a target positive label mask word vector according to a hierarchical relationship of positive label mask word vectors on a plurality of positive label mask characters;
[0106] Step S202, determining a cross-layer ranking loss according to a first similarity between the target positive label mask word vector and the positive sub-label word vector, and a second similarity between the target positive label mask word vector and the negative sub-label word vector;
[0107] Step S203, performing a weighted combination of the cross-layer ranking loss, the positive label contrast loss, and the negative label contrast loss to obtain a prompt label contrast learning loss;
[0108] Step S204, determining the classification task loss by comparing the learning loss and the prediction result according to the prompt label.
[0109] In this embodiment, the hint label contrast learning loss also includes a cross-layer ranking loss. When the positive label contrast loss and the negative label contrast loss are calculated, the cross-layer ranking loss is calculated, and then the positive label contrast loss, the negative label contrast loss and the cross-layer ranking loss are integrated to determine the hint label contrast learning loss.
[0110] The purpose of cross-layer ranking loss is to align the embedding vector of the label mask word with the representation of its sub-label, and it is expected that the semantic similarity between the high-level label (i.e. the target positive label mask word vector) and its positive sub-label (i.e. the positive sub-label word vector) is greater than the similarity with the remaining sub-labels and the same-level labels (the remaining sub-labels and the same-level labels are negative sub-label word vectors). The formula is as follows:
[0111]
[0112] in, Represents the representation vector of the negative sub-label of the positive label at layer l, Represents the negative label representation vector of the l+1th layer that does not belong to the positive label of the lth layer.
[0113] The weighted combination of positive label contrast loss, negative label contrast loss and cross-layer ranking loss suggests that the label contrast learning loss is expressed as:
[0114]
[0115] Among them, α and β are hyperparameters. Based on the above formula, the prompt label comparison loss value of the sample data in the training data is calculated.
[0116] See also Figure 3 In some embodiments, the hierarchical text classification model training method of the embodiment of the present application may also include but is not limited to steps S301 to S303:
[0117] Step S301, mapping the sample label text to an identification word to obtain an identification representation label of the sample label, wherein the sample label includes the true label text and negative label text of the sample data;
[0118] Step S302, using a preset hierarchical structure representation form to perform hierarchical flattening representation processing on the identification representation tags with hierarchical structure information to obtain a tag sequence;
[0119] Step S303, inputting the label sequence into the feature representation layer of the hierarchical text classification model to obtain a sample label vector, wherein the sample label vector includes a true label vector and a negative label vector.
[0120] In this embodiment, in the process of calculating the prompt label contrast loss, a representation vector of the sample label text is required, that is, the true label vector and the negative label vector for comparison. The sample label encoding method in this embodiment is as follows:
[0121] In order to avoid the influence of overlapping interactions between label names and text words, a unique special identifier is assigned to each label, such as "L0" for the first label category, and all identifiers are added to the vocabulary of model training. In the process of encoding labels, the sample label text (real label text or negative label text) can be mapped to the identifier based on the vocabulary of model training, and the label can be replaced by the identifier or the identifier is added to the label to obtain the identifier representation label of the sample label.
[0122] In order to incorporate the hierarchical information into the label representation, the parent-child hierarchy of the mapped identification representation label with the hierarchical information can be flattened to form a label sequence, as shown below:
[0123] Q={[CLS]W[SEP]W f [SEP]{W c}[SEP]};
[0124] Among them, W f Indicates the parent tag identifier of the current tag identifier W, {W c} represents all its child tags. When a tag has no parent tag or child tag, "Root" and "None" are used respectively.
[0125] The label sequence composed of the above label words is input into the BERT model shared with the text, and the embedding vector at the W identification position is taken as the representation vector r of the current label.
[0126] See also Figure 4 In some embodiments, the negative label text in step S301 may be obtained by including but not limited to steps S401 to S402:
[0127] Step S401, sorting the hierarchical label categories output by the hierarchical text classification model according to the order of confidence scores from large to small;
[0128] Step S402: taking the top preset number of hierarchical label categories as negative label texts.
[0129] In this embodiment, in the positive label contrast learning process, a Top-K difficult negative sample dynamic sampling algorithm can be used to efficiently select negative samples for contrast learning. Specifically, at each level, all label categories output by the model are sorted in descending order according to the confidence scores output by the mapper, and the top K negative labels are selected as difficult negative samples to participate in contrast learning. By selecting labels with higher confidence scores as negative samples for positive label contrast learning, the training model can identify incorrect labels similar to correct labels, thereby improving the accuracy of model text classification.
[0130] See also Figure 5 In some embodiments, step S204 may also include but is not limited to steps S501 to S504:
[0131] Step S501, based on the cross entropy loss function, determining the consistency loss according to the hierarchical relationship prediction score of the hierarchical label category, wherein the hierarchical relationship prediction score represents the prediction score of the hierarchical text classification model on whether the hierarchical label category conforms to the true hierarchical relationship;
[0132] Step S502, based on the cross entropy loss function, determining the classification correctness loss according to the semantic classification prediction score of the hierarchical label category, wherein the semantic classification prediction score represents the prediction score of whether the hierarchical text classification model conforms to the true label;
[0133] Step S503, determining the auxiliary task loss according to the consistency loss and the classification correctness loss;
[0134] Step S504, determining the classification task loss according to the prompt label contrast learning loss, the auxiliary task loss, the model overall classification loss and the masked language model loss.
[0135] In this embodiment, hierarchical relationship consistency is an important evaluation indicator of the hierarchical text classification algorithm, which requires the model to correctly identify the correlation between categories and maintain the consistency of this relationship when predicting labels at all levels. In the related art, hierarchical text classification is usually converted into a flat multi-label text classification task, but this method simplifies the problem while also weakening the inherent hierarchical connection between labels. Therefore, the embodiment of the present application performs an auxiliary task of self-awareness of the label hierarchy, calculates the consistency and correctness loss of candidate labels, and enables the model to learn the hierarchical relationship between label categories based on the auxiliary loss, as follows:
[0136] A simple feed-forward neural network is used on the [CLS] position of the BERT model to score the consistency and accuracy of the candidate label set (i.e., hierarchical label categories, including multiple labels with hierarchical relationships) output by the mapper.
[0137] Based on the cross entropy loss function, the consistency loss of the candidate labels is calculated. The calculation formula is as follows:
[0138]
[0139] Among them, p con It represents the prediction score of the hierarchical text classification model for the candidate tags that meet the hierarchical relationship. When all the tags in the candidate tag set happen to meet the hierarchical relationship, the true value y con =1, otherwise y con =0.
[0140] Based on the cross entropy loss function, the classification correctness loss of the candidate label is calculated. The calculation formula is as follows:
[0141]
[0142] Among them, p cor It indicates the prediction score of the hierarchical text classification model on whether the candidate label is the true label. When all labels in the candidate label set happen to be the true labels of the sample, the true value y cor =1, otherwise y cor =0.
[0143] The auxiliary task loss is determined based on the consistency loss and the classification correctness loss. The auxiliary task loss is expressed as follows:
[0144]
[0145] After getting the prompt label contrast learning loss and auxiliary task loss The model's cross entropy classification loss can be further combined and masked language model loss Determine the classification task loss. The classification task loss is calculated as follows:
[0146]
[0147] In the formula, λ1 and λ2 are hyperparameters used to adjust the weights of different losses.
[0148] To minimize the overall loss As the goal, iterative training is performed to continuously update and optimize the parameters of the model until the model converges. In this embodiment, a pre-trained model consistent with the backbone network can be selected.
[0149] According to some embodiments of the present application, combined Fig.11 , the overall training process of the hierarchical text classification model of the embodiment of the present application is explained.
[0150] S11, labeling the training dataset using a predefined label architecture;
[0151] S12, using the constructed dual prompt template to perform form conversion on the sample data in the training data set;
[0152] S13, inputting the converted sample data into the constructed hierarchical text classification model (i.e., multi-label text classification model) to obtain the model's prediction results, positive label mask word vectors, and negative label mask word vectors;
[0153] S14. Calculate the multi-task classification loss of the model based on the prediction results, the positive label mask word vectors, and the negative label mask word vectors, including the overall multi-label classification task loss of the model, the mask language model loss, the prompt label contrast learning loss, and the auxiliary loss of label level self-awareness;
[0154] S15. Perform gradient back propagation according to the multi-task classification loss to update the parameters of the hierarchical text classification model;
[0155] S16, judging whether the hierarchical text classification model has converged, if not, returning to step S12 to continue updating the model, if converged, executing step 17;
[0156] S17. Output the trained hierarchical text classification model.
[0157] Figure 6 is an optional flowchart of the hierarchical text classification method provided in the embodiment of the present application. Figure 6The method may include but is not limited to steps S601 to S603:
[0158] Step S601, obtaining the text to be inferred;
[0159] Step S602, converting the inference text into a form according to the dual prompt template to obtain an input text;
[0160] Step S603, inputting the input text into the hierarchical text classification model to obtain a hierarchical text prediction result, and determining the hierarchical classification result of the text to be inferred according to the hierarchical text prediction result, wherein the hierarchical text classification model is obtained by the hierarchical text classification model training method as in the previous embodiment.
[0161] In this embodiment, the text to be inferred is filled into the dual prompt template, the input text is reconstructed, and the trained hierarchical text classification model is used for prediction to obtain the hierarchical text prediction result. The hierarchical text prediction result includes multiple hierarchical label categories and the confidence score of each hierarchical label category. The embodiment of the present application can further determine the hierarchical classification result of the text to be inferred based on the hierarchical text prediction result. For example, the hierarchical label category with the highest confidence score of the hierarchical text prediction result is taken as the hierarchical classification result of the text to be inferred.
[0162] See also Figure 7 In some embodiments, the step of determining the hierarchical classification result of the text to be inferred according to the hierarchical text prediction result in step S603 may include, but is not limited to, steps S701 to S705:
[0163] Step S701, determining whether the highest confidence score in the hierarchical text prediction result is greater than a preset score;
[0164] Step S702, when the highest confidence score in the hierarchical text prediction result is greater than a preset score, the hierarchical classification result of the text to be inferred is determined according to the hierarchical text prediction result;
[0165] Step S703, when the highest confidence score in the hierarchical text prediction result is less than or equal to the preset score, the hierarchical label categories with the second preset number of confidence scores in the hierarchical text prediction result are taken as candidate hierarchical labels;
[0166] Step S704, filling the context learning prompt template of the large language model according to the candidate level labels to obtain the text classification question text;
[0167] Step S705: input the text classification question text into the large language model to obtain a text classification answer, and analyze and extract the text classification answer to obtain a hierarchical classification result of the text to be inferred.
[0168] For details, please refer to Fig.12 , fill the text to be inferred into the dual prompt template to obtain the input text, input the input text into the hierarchical text classification model (i.e., the multi-label classification small model) of the previous embodiment for preliminary prediction, if the highest confidence score in the hierarchical classification result output by the small model is greater than the threshold S (the value of S can be determined according to the application scenario and the classification effect of the small model), it indicates that the text to be inferred is a simple text, and the hierarchical label category with the highest confidence score predicted by the small model can be directly used as the predicted label of the text to be inferred, or the hierarchical label category with a confidence score predicted by the small model greater than the threshold F (for example, 0.5) can be used as the predicted label of the text to be inferred. If the highest confidence score in the hierarchical classification result output by the small model is less than or equal to the threshold S, it indicates that the text to be inferred is a difficult text, and the large language model reorders and selects the prediction results of the small model to determine the predicted label of the text to be inferred. The specific process is as follows:
[0169] Design the instruction prompt template (i.e., context learning prompt template) for context learning of large language models, which consists of task description, several examples, and queries. An example of the context learning prompt template is shown in Table 1:
[0170] Table 1 Contextual learning prompt template
[0171]
[0172]
[0173] According to the hierarchical label categories and confidence scores output by the classification model, the score of each hierarchical label (i.e., a path in the corresponding label tree) is calculated as: the average confidence score of each label node in the hierarchical label. The C hierarchical labels with the highest scores (the value of C depends on the application scenario and the classification effect of the small model) are obtained as the candidate label set for the sample.
[0174] Each sample in the training set is inferred by a small model, and the candidate label set of each sample is obtained according to the above steps, and the order of the candidate labels is randomly shuffled. The training set is used as a sample library, and each sample is constructed into an example according to the format specified in the template. Among them, "text" is the text content of the example, "options" are C candidate label sets, and "answers" are the correct labels in the "options".
[0175] For each candidate tag of the query question, based on the principle of maximizing text similarity, examples are selected from the training samples whose true tags are the tags to help the model understand the task. Specifically,
[0176] Based on the BGE vector model, the feature vector of each text in the example library is obtained and saved.
[0177] The feature vector of the query text is obtained based on the BGE vector model.
[0178] Based on the cosine similarity function, the text similarity between the query text and all samples under a certain label is calculated.
[0179] The training samples with the highest similarity are selected as examples in the contextual learning prompt template.
[0180] Fill the query question, candidate tags, and examples into the prompt template to form a structured input.
[0181] Use pre-trained large language models, such as TeleChat, to continue and answer input text.
[0182] Extract answers from the responses generated by the big prediction model and normalize the model output.
[0183] According to some embodiments of the present application, the embodiments of the present application have the following beneficial effects:
[0184] The hierarchical text classification method based on "dual prompt fine-tuning" in the embodiment of the present application can alleviate the confusion between hierarchical labels. Specifically, the method constructs a dual prompt template, and executes an original hierarchical-aware prompt label comparison learning algorithm based on the prompt template, fully extracts the feature embedding of the label prompt words, and then maps the embedding on the label mask bit to the category name based on the "Multi-Verbalizers" framework. This method can not only encourage the model to identify the correct label, but also identify the wrong label that tends to be confused with the positive label, thereby helping to extract semantic features that are highly discriminative and rich in hierarchical information.
[0185] The embodiment of the present application introduces a cross-level ranking loss in the prompt label contrast learning. The ranking loss is based on the structural association relationship of the hierarchical labels and models the semantic distance relationship between labels in the feature space. The ranking loss encourages the semantic similarity between the parent category and the child category to be higher than the similarity with other categories in the next level, which effectively alleviates the problem of difficulty in aligning hierarchical labels.
[0186] The embodiment of the present application is based on the auxiliary task loss update model of self-awareness of the label hierarchy, so that the model maintains the best perception ability on the overall hierarchical structure. The auxiliary task does not simply merge the encoding of the hierarchical structure with the text semantic features, but uses a layer of neural network to learn and capture the consistency and correctness of each candidate label path. This auxiliary task helps the model deeply internalize the structural hierarchical knowledge, improves the hierarchical consistency of the prediction results, and enables the model to more accurately identify the category attribution of text at different granularity levels.
[0187] The embodiment of the present application utilizes the in-context learning capability of the large model as a sorter to re-evaluate and sort the output labels of the small model on difficult samples with low confidence and high perplexity, thereby improving the classification performance on difficult samples. A sample selection strategy based on maximizing text similarity is developed to achieve more accurate and efficient classification learning.
[0188] See also Figure 8 The present application also provides a hierarchical text classification model training system including:
[0189] The first module is used to obtain a training dataset annotated according to a predefined label architecture;
[0190] The second module is used to perform form conversion on the sample data in the training data set according to the dual prompt template to obtain the prompt text, wherein the dual prompt template includes text data, a positive hierarchical relationship string and a negative hierarchical relationship string, the positive hierarchical relationship string includes a plurality of positive label mask characters representing the hierarchical relationship, and the negative hierarchical relationship string includes a plurality of negative label mask characters representing the hierarchical relationship;
[0191] The third module is used to input the text data in the prompt text into the initialized hierarchical text classification model for forward transmission, and obtain the positive label mask word vector on the positive label mask character, the negative label mask word vector on the negative label mask character and the prediction result, wherein the prediction result includes the hierarchical label category and the confidence score of the hierarchical label category;
[0192] The fourth module is used to perform positive label contrast learning on positive label mask characters according to the positive label mask word vector, the true label vector and the negative label vector of the text data to obtain the positive label contrast loss; perform negative label contrast learning on negative label mask characters according to the negative label mask word vector, the true label vector and the negative label vector of the text data to obtain the negative label contrast loss; determine the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result;
[0193] The fifth module is used to update the parameters of the hierarchical text classification model according to the classification task loss to obtain a trained hierarchical text classification model.
[0194] In this embodiment, the first module is mainly used to construct a training data set; the second module is mainly used to perform form conversion processing on the text of the training data set using a dual prompt template; the third module is mainly used to execute the forward pass process of the model to obtain relevant data of the forward pass process; the fourth module is mainly used to calculate the classification task loss of the model; the fifth module is mainly used to execute the reverse update process of the model to adjust the model parameters.
[0195] It can be understood that the contents of the above-mentioned hierarchical text classification model training method embodiment are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above-mentioned hierarchical text classification model training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned hierarchical text classification model training method embodiment.
[0196] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned hierarchical text classification model training method or hierarchical text classification method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0197] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0198] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0199] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the hierarchical text classification model training method or hierarchical text classification method of the embodiment of this application;
[0200] Input / output interface 903, used to implement information input and output;
[0201] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0202] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0203] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0204] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned hierarchical text classification model training method or hierarchical text classification method.
[0205] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0206] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0207] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0208] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0209] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0210] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0211] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0212] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.
[0213] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0216] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A hierarchical text classification model training method, characterized in that: The following steps are involved: Obtain a training dataset annotated according to a predefined labeling architecture; The sample data in the training data set is converted into a form according to a dual prompt template to obtain a prompt text, wherein the dual prompt template includes text data, a positive hierarchical relationship string and a negative hierarchical relationship string, the positive hierarchical relationship string includes a plurality of positive label mask characters representing the hierarchical relationship, and the negative hierarchical relationship string includes a plurality of negative label mask characters representing the hierarchical relationship; Inputting the text data in the prompt text into the initialized hierarchical text classification model for forward transmission, obtaining positive label mask word vectors on positive label mask characters, negative label mask word vectors on negative label mask characters and prediction results, wherein the prediction results include hierarchical label categories and confidence scores of hierarchical label categories; According to the positive label mask word vector, the true label vector and the negative label vector of the text data, positive label contrast learning is performed on the positive label mask character to obtain the positive label contrast loss; According to the negative label mask word vector, the true label vector and the negative label vector of the text data, negative label contrast learning is performed on the negative label mask character to obtain a negative label contrast loss; Determine the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result; The parameters of the hierarchical text classification model are updated according to the classification task loss to obtain a trained hierarchical text classification model.
2. The hierarchical text classification model training method according to claim 1, characterized in that: The step of determining the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result comprises the following steps: Determine a positive sub-label word vector and a negative sub-label word vector of a target positive label mask word vector according to a hierarchical relationship of the positive label mask word vectors on a plurality of positive label mask characters; Determine a cross-layer ranking loss according to a first similarity between the target positive label mask word vector and the positive sub-label word vector, and a second similarity between the target positive label mask word vector and the negative sub-label word vector; Performing a weighted combination of the cross-layer ranking loss, the positive label contrast loss, and the negative label contrast loss to obtain a hint label contrast learning loss; The classification task loss is determined based on the prompt label comparison learning loss and the prediction result.
3. The hierarchical text classification model training method according to claim 1, characterized in that: The hierarchical text classification model training method also includes the following steps: Performing identification word mapping on the sample label text to obtain an identification representation label of the sample label, wherein the sample label includes a true label text and a negative label text of the sample data; Using a preset hierarchical structure representation form to perform hierarchical flattening representation processing on the identification representation labels with hierarchical structure information to obtain a label sequence; The label sequence is input into the feature representation layer of the hierarchical text classification model to obtain a sample label vector, wherein the sample label vector includes a true label vector and a negative label vector.
4. The hierarchical text classification model training method according to claim 3, characterized in that: The negative label text is obtained by the following steps, including: Sorting the hierarchical label categories output by the hierarchical text classification model according to the confidence scores from large to small; The top preset number of hierarchical label categories are used as negative label texts.
5. The hierarchical text classification model training method according to claim 2, characterized in that: Determining the classification task loss according to the prompt label comparison learning loss and the prediction result includes the following steps: Based on the cross entropy loss function, the consistency loss is determined according to the hierarchical relationship prediction score of the hierarchical label category, wherein the hierarchical relationship prediction score represents the prediction score of the hierarchical text classification model on whether the hierarchical label category conforms to the true hierarchical relationship; Based on the cross entropy loss function, determining the classification correctness loss according to the semantic classification prediction score of the hierarchical label category, wherein the semantic classification prediction score represents the prediction score of the hierarchical text classification model on whether the hierarchical label category conforms to the true label; Determining an auxiliary task loss according to the consistency loss and the classification correctness loss; The classification task loss is determined according to the prompt label contrast learning loss, the auxiliary task loss, the model overall classification loss and the masked language model loss.
6. A hierarchical text classification method, characterized in that: The following steps are involved: Get the text to be inferred; Performing form conversion on the text to be inferred according to the dual prompt template to obtain an input text; The input text is input into a hierarchical text classification model to obtain a hierarchical text prediction result, and the hierarchical classification result of the text to be inferred is determined according to the hierarchical text prediction result, wherein the hierarchical text classification model is obtained by the hierarchical text classification model training method according to any one of claims 1 to 5.
7. The hierarchical text classification method according to claim 6, characterized in that: The hierarchical text prediction result includes multiple hierarchical label categories and confidence scores of each hierarchical label category. Determining the hierarchical classification result of the text to be inferred according to the hierarchical text prediction result includes the following steps: Determining whether the highest confidence score in the hierarchical text prediction result is greater than a preset score; When the highest confidence score in the hierarchical text prediction result is greater than a preset score, the hierarchical classification result of the text to be inferred is determined according to the hierarchical text prediction result; When the highest confidence score in the hierarchical text prediction result is less than or equal to a preset score, the hierarchical label categories with the second-preset number of confidence scores in the hierarchical text prediction result are taken as candidate hierarchical labels; Filling the context learning prompt template of the large language model according to the candidate level labels to obtain a text classification question text; The text classification question text is input into a large language model to obtain a text classification answer, and the text classification answer is analyzed and extracted to obtain a hierarchical classification result of the text to be inferred.
8. A hierarchical text classification model training system, characterized in that: include: The first module is used to obtain a training dataset annotated according to a predefined label architecture; The second module is used to perform form conversion on the sample data in the training data set according to the dual prompt template to obtain a prompt text, wherein the dual prompt template includes text data, a positive hierarchical relationship string and a negative hierarchical relationship string, the positive hierarchical relationship string includes a plurality of positive label mask characters representing the hierarchical relationship, and the negative hierarchical relationship string includes a plurality of negative label mask characters representing the hierarchical relationship; The third module is used to input the text data in the prompt text into the initialized hierarchical text classification model for forward transmission, and obtain the positive label mask word vector on the positive label mask character, the negative label mask word vector on the negative label mask character and the prediction result, wherein the prediction result includes the hierarchical label category and the confidence score of the hierarchical label category; The fourth module is used to perform positive label contrast learning on the positive label mask characters according to the positive label mask word vector, the true label vector and the negative label vector of the text data to obtain the positive label contrast loss; perform negative label contrast learning on the negative label mask characters according to the negative label mask word vector, the true label vector and the negative label vector of the text data to obtain the negative label contrast loss; determine the classification task loss according to the positive label contrast loss, the negative label contrast loss and the prediction result; The fifth module is used to update the parameters of the hierarchical text classification model according to the classification task loss to obtain a trained hierarchical text classification model.
9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the hierarchical text classification model training method described in any one of claims 1 to 5 or the hierarchical text classification method described in any one of claims 6 to 7 are realized.
10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the hierarchical text classification model training method described in any one of claims 1 to 5 or the hierarchical text classification method described in any one of claims 6 to 7.
Citation Information
Cited By
Hierarchical text classification method and device based on hybrid experts
CN120541225A
