Text multi-label classification method and device, electronic equipment and storage medium

Abstract

The invention provides a text multi-label classification method and device, electronic equipment and a storage medium. The text multi-label classification method comprises the following steps: inputting a technical field and labels needing to be classified into a large language model, so that the large language model generates a definition corresponding to each label in the technical field; annotating the literature data by using a large language model to obtain annotated data; loRA fine tuning is carried out on a large language model by using the annotation data to obtain a fine-tuned model, and the fine-tuned model is used for multi-label classification; and based on the definition of each label, performing multi-label classification on the text by utilizing the fine-tuned model. According to the text multi-label classification method, technical labels can be accurately understood, knowledge contained in literatures can be learned, the classification effect of the literature data multi-label classification task is good, and the classification accuracy and the classification effect of the literature data multi-label classification task are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a text multi-label classification method, a classification device, an electronic device and a storage medium. Background Art

[0002] Emerging industries and future industries are booming based on the application of new technologies, and have a global impact on the progress of the national economy, social development, and the optimization and upgrading of the industrial structure. Standardization plays a fundamental and leading key role in the development of these new industries. Carrying out new industry standardization pilot projects is of great significance for promoting the high-quality development of new industries and accelerating the construction of a modern industrial system. Journal papers and other technical documents are indispensable materials for the forward-looking layout of future industry standard research. In order to use technical documents to conduct standardization research on key technologies of future industries, how to classify technical documents according to key technologies has become an urgent problem to be solved. In order to promote the development of future industry standardization work, it is necessary to classify technical documents related to future industries according to key technologies by multi-label. The multi-label classification method of related technologies makes it difficult to accurately understand technical labels and learn the knowledge contained in the documents, resulting in poor results when performing multi-label classification tasks of document data. The above background technical information does not necessarily constitute prior art. Summary of the invention

[0003] The purpose of this application is to provide a text multi-label classification method, classification device, electronic device and storage medium, which can improve the accuracy and classification effect of multi-label classification tasks of document data.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, the present application provides a text multi-label classification method, comprising:

[0006] Input the technical field and the labels to be classified into the large language model, so that the large language model generates the definition of each label under the corresponding technical field;

[0007] Use a large language model to annotate the document data to obtain annotated data;

[0008] Using the labeled data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model, wherein the fine-tuned model is used for multi-label classification;

[0009] Based on the definition of each label, the text is classified into multiple labels using the fine-tuned model.

[0010] In some embodiments, the step of using the annotated data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model includes:

[0011] Inputting Chinese document abstract data into the large language model, and translating the Chinese document abstract data into at least two foreign languages ​​through the large language model as data for LoRA fine-tuning;

[0012] Based on the explanation of the technical label in the label definition and the at least two foreign languages, the large language model is fine-tuned using LoRA to obtain a fine-tuned model.

[0013] In some embodiments, the method of performing multi-label classification on the text based on the definition of each label and using the fine-tuned model includes:

[0014] Using the fine-tuned model to predict the text data, predict the Chinese document abstract data and the data sets of at least two foreign languages ​​respectively, and obtain the final answer by voting;

[0015] When two or more languages ​​in the voting plan consider a label to be correct, the label will be added to the final answer.

[0016] In some embodiments, the interpretation of the technical label in the definition of the combined label and the at least two foreign languages ​​to perform LoRA fine-tuning on the large language model includes:

[0017] constructing a prompt template in combination with the explanation of the technical label in the definition of the label and the at least two foreign languages;

[0018] The prompt template is input into the large language model for multilingual LoRA fine-tuning.

[0019] In some embodiments, the multi-label classification of text includes:

[0020] For a given literature summary dataset and a label set , find the tags related to the literature summary dataset, where Describes the number of abstracts, where Indicates the number of tags, each document Both Relevance, indicating whether the tag is related to the document.

[0021] In some embodiments, the method further comprises:

[0022] Micro-accuracy and Micro-F1 score are used to evaluate the classification results; Micro-accuracy is the overall average result of the precision of all labels; Micro-F1 score is an indicator that comprehensively considers precision and recall, and is calculated by micro-averaging the precision and recall of all labels in multi-label classification.

[0023] In some embodiments, the large language model comprises a GLM-4 model.

[0024] In a second aspect, the present application provides a text multi-label classification device, comprising:

[0025] A label definition generation module, used to input the technical field and the labels to be classified into the large language model, so that the large language model generates the definition of each label under the corresponding technical field;

[0026] An annotation module is used to annotate the document data using a large language model to obtain annotated data;

[0027] A fine-tuning module, used to perform LoRA fine-tuning on the large language model using the labeled data to obtain a fine-tuned model, wherein the fine-tuned model is used for multi-label classification;

[0028] The classification module is used to perform multi-label classification on the text based on the definition of each label using the fine-tuned model.

[0029] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; when the processor executes the running program stored in the memory, the text multi-label classification method described in any embodiment of the present application is implemented.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text multi-label classification method described in any embodiment of the present application.

[0031] The text multi-label classification method provided by the present application inputs the technical field and the labels to be classified into a large language model so that the large language model generates the definition of each label under the corresponding technical field, uses the large language model to annotate the document data to obtain annotated data, uses the annotated data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model, and the fine-tuned model is used for multi-label classification. Based on the definition of each label, the text is classified into multi-labels using the fine-tuned model, which can accurately understand the technical labels and learn the knowledge contained in the documents. The classification effect of the document data multi-label classification task is good, which greatly improves the classification accuracy and classification effect of the document data multi-label classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a text multi-label classification method provided by an embodiment of the present application is shown.

[0033] Figure 2 A schematic diagram of the structure of a large language model provided by an embodiment of the present application is shown.

[0034] Figure 3 A schematic diagram of the answer voting steps provided by an embodiment of the present application is shown.

[0035] Figure 4 A structural block diagram of a text multi-label classification device provided by an embodiment of the present application is shown.

[0036] Figure 5 A structural block diagram of an electronic device provided by an embodiment of the present application is shown.

[0037] Figure 6 A schematic diagram of a computer-readable storage medium provided by an embodiment of the present application is shown.

[0038] Figure 7 A GLM-4 label interpretation diagram provided by one embodiment of the present application is shown.

[0039] Figure 8 A data translation diagram provided by an embodiment of the present application is shown.

[0040] Fig. 9 A LoRA fine-tuning prompt diagram provided by an embodiment of the present application is shown.

[0041] Fig.10 A label diagram of key technologies for obtaining a large model provided by an embodiment of the present application is shown.

[0042] Fig.11 A large language model document abstract annotation diagram provided by an embodiment of the present application is shown.

[0043] Fig.12 A diagram showing experimental results provided by an embodiment of the present application is shown.

[0044] Fig.13 A diagram showing ablation experiment results provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] In order to explain the features and technical contents of the embodiments of the present application in more detail, the technical solutions of the present application will be further described below in combination with the drawings and specific embodiments of the specification. The drawings are only used for reference and are not used to limit the scope of the embodiments of the present application.

[0046] Unless otherwise defined, all technical and scientific terms used in the examples of this application shall have the same meaning as those generally understood by those skilled in the art. The use of terms is only for the purpose of describing the examples of this application and is not intended to limit the application in any way.

[0047] In the following description, the “some embodiments” mentioned represent a subset of all possible embodiments, but it should be understood that “some embodiments” may be the same or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that if terms such as “first, second, third” appear in the embodiments of the present application, these terms are only used to distinguish between similar objects and do not represent a specific order or sequence. Therefore, terms such as “first, second, third” may be used interchangeably where permitted, and the embodiments of the present application do not have to be performed strictly in the order described.

[0048] In the related art, the multi-label classification problem is converted into multiple single-label classification sub-problems through the problem conversion method, and then these sub-problems are solved using mature single-label classification algorithms. For example, an independent binary classifier is constructed for each label to solve the multi-label classification problem, but constructing a separate binary classifier for each label makes it ignore the relationship between labels and cannot learn the association between labels well. Alternatively, by considering a small random subset of labels and learning a single-label classifier for the power set of this subset, the single-label classifier is applied to the subtask to consider the correlation between labels, which achieves better performance than the BR method, but the randomly selected subset may not fully cover all the characteristics of the label relationship, and the integration method increases the computational complexity.

[0049] In the related technology, pairwise classification is first used to induce binary preference relations from suitable training data, and then a ranking program is used to derive rankings from the preference relations for multi-label classification. This method can produce a ranking relationship for labels, which has advantages in some scenarios where ranking results are required. However, the pairwise comparison process is computationally intensive and has a strong dependence on data. When the data quality is poor, the ranking results may be inaccurate.

[0050] In order to at least partially solve the problem of poor classification results in multi-label classification tasks for document data in related technologies, the method of the embodiment of the present application uses LoRA to fine-tune GLM4 for multi-label classification of document texts, uses a large language model to interpret domain technical labels, and deepens the large language model's understanding of labels; uses multilingual data to perform LoRA fine-tuning on GLM4, allowing the large language model to understand technical documents from multiple language perspectives and obtain the final answer through voting. Experimental verification was conducted on the constructed document multi-label classification dataset, and compared with the baseline model, Acc increased by 37.43 percentage points and F1 increased by 14.54 percentage points.

[0051] The text multi-label classification method of the embodiment of the present application first annotates the document data. In the related art, manual data annotation requires a lot of manpower costs, especially when performing multi-label annotation. The length of the document abstract is long and obscure, which increases the difficulty of annotation. Therefore, this paper uses a large language model (LLM) to assist in label annotation. With the help of the massive knowledge of the large language model, errors in the annotation process are reduced, and the annotation workload is reduced. The combined use of multiple large language models combined with manual annotation effectively reduces the impact of the "cognitive" bias of the large language model and ensures the accuracy of annotation. After annotating the data, this paper uses the annotated data to perform LoRA fine-tuning on the large language model for multi-label classification. The accuracy of multi-label classification of the large language model is improved by using the large language model to annotate the technical labels in the relevant technical field and using multi-language data to fine-tune the training of the large language model. Specifically, the beneficial technical effects achieved by the method of the embodiment of the present application include: proposing a method combined with a large language model to assist in labeling, which can effectively reduce the cost of manual labeling during multi-label labeling and improve the efficiency and accuracy of labeling; using domain technology label interpretation and multilingual training data to perform LoRA fine-tuning on the large language model, and voting to obtain the final multi-label classification result, thereby improving the accuracy of multi-label classification.

[0052] The development of deep learning has promoted the application of deep neural networks in multi-label text classification tasks and achieved good results. Multi-label text classification methods mainly include machine learning-based methods and deep learning-based methods. Among them, machine learning methods can be divided into problem conversion methods and algorithm adaptation methods according to different problem-solving angles.

[0053] The large language model has powerful semantic understanding and context perception capabilities and a massive knowledge reserve. These advantages enable the large language model to achieve better classification results when there are few samples and complex language environments. Therefore, this paper uses the large language model for multi-label classification of document texts.

[0054] The multi-label text classification task of documents can be defined as: using the content of a given document to perform multi-label classification on the document. In this paper, for a given document abstract dataset ,in Describes the number of literature abstracts and a tag set ,in Indicates the number of tags, each document Both Correlation indicates whether the tag is related to the document. Multi-label text classification is to build a prediction model to find tags related to new text.

[0055] refer to Figure 1As shown, an embodiment of the present application provides a text multi-label classification method, which may include:

[0056] S10. Input the technical field and the labels to be classified into the large language model, so that the large language model generates the definition of each label under the corresponding technical field.

[0057] S20. Use the large language model to annotate the document data to obtain annotated data.

[0058] S30. Use the labeled data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model, where the fine-tuned model is used for multi-label classification.

[0059] Exemplarily, the using the annotated data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model includes:

[0060] Inputting Chinese document abstract data into the large language model, and translating the Chinese document abstract data into at least two foreign languages ​​through the large language model as data for LoRA fine-tuning;

[0061] Combined with the explanation of the technical label in the label definition and the at least two foreign languages, the large language model is fine-tuned by LoRA to obtain a fine-tuned model. The at least two foreign languages ​​may include, for example, English, Japanese, etc.

[0062] Exemplarily, based on the definition of each label, the fine-tuned model is used to perform multi-label classification on the text, including:

[0063] Using the fine-tuned model to predict the text data, predict the Chinese document abstract data and the data sets of at least two foreign languages ​​respectively, and obtain the final answer by voting;

[0064] When two or more languages ​​in the voting plan consider a label to be correct, the label will be added to the final answer.

[0065] Exemplarily, combining the explanation of the technical label in the label definition and the at least two foreign languages, performing LoRA fine-tuning on the large language model includes:

[0066] constructing a prompt template in combination with the explanation of the technical label in the definition of the label and the at least two foreign languages;

[0067] The prompt template is input into the large language model for multilingual LoRA fine-tuning.

[0068] The method of this embodiment combines the LoRA and GLM-4 models, which can achieve the following beneficial technical effects:

[0069] Reduce computational overhead: LoRA is a lightweight fine-tuning method that adjusts the weights of the model by inserting low-rank matrices in specific layers of the pre-trained model instead of fine-tuning the entire model. This can significantly reduce the consumption of computing resources, especially on large-scale models such as GLM-4. Compared with traditional fine-tuning methods, LoRA achieves more efficient training and inference by optimizing a small number of parameters.

[0070] Improve customization capabilities: Through LoRA fine-tuning, the pre-trained GLM-4 can be customized according to specific task or domain requirements. For example, in a specific field, the model may need to better understand specific terms and label definitions, and LoRA can make GLM-4 better adapt to the knowledge requirements of these specific fields through a small amount of adjustment.

[0071] Save storage space: The LoRA method does not need to save the fine-tuned version of the entire model, only the low-rank adaptation matrix, which greatly reduces the storage overhead. This is especially important for large-scale language models (such as GLM-4), especially when different fine-tuning is required for multiple domains.

[0072] Enhance the generalization ability of the model: LoRA can avoid overfitting and improve the generalization ability of the model on different tasks by fine-tuning part of the model's parameters without changing the entire network structure. Especially for tasks with rich domain labels, LoRA can help GLM-4 share knowledge between multiple tasks while maintaining flexibility and adaptability.

[0073] Optimize the reasoning process: During the LoRA fine-tuning process, the model parameter adjustment is achieved through a low-rank matrix, which not only optimizes the training process, but also reduces the number of parameters during reasoning, making reasoning more efficient. When LoRA and GLM-4 are combined, the generated label definition and input parts can not only improve the accuracy of task completion, but also maintain low latency during reasoning.

[0074] In summary, the combination of LoRA and GLM-4 can enhance the model's customization, generalization, and reasoning efficiency without significantly increasing computing and storage overhead, and can provide more efficient and accurate text multi-label classification.

[0075] S40: Based on the definition of each label, use the fine-tuned model to perform multi-label classification on the text.

[0076] Exemplarily, multi-label classification of text includes:

[0077] For a given literature summary dataset and a label set , find the tags related to the literature summary dataset, where Describes the number of abstracts, where Indicates the number of tags, each document Both Relevance, indicating whether the tag is related to the document.

[0078] Exemplarily, the method may further include:

[0079] S50, use Micro-accuracy and Micro-F1 score to evaluate the classification results; Micro-accuracy is the overall average result of the precision of all labels; Micro-F1 score is an indicator that comprehensively considers precision and recall, and is calculated by micro-averaging the precision and recall of all labels in multi-label classification.

[0080] Exemplarily, the large language model includes but is not limited to a GLM-4 model.

[0081] The method of the embodiment of the present application uses a large language model to predict the labels of the document text. Specifically, the technical field and key technology labels are input into GLM-4, and GLM-4 is used to deeply interpret the key technology labels of the document. Through the powerful semantic understanding ability of the model, the connotation definition of the label is accurately analyzed, and the relationship between the technical field and the label is analyzed. The abstract of the document is then input into GLM-4. With the help of the multi-language processing ability of the model, the abstract of the document is translated into foreign language data such as English and Japanese data. Based on the above processing combined with the label definition and the translation data, a prompt template is constructed, which is input into GLM-4 for multi-language LoRA fine-tuning. Model such as Figure 2 The texts or documents mentioned in this article include but are not limited to papers, patent texts, etc. Papers include but are not limited to journal articles, master's and doctoral dissertations of universities, etc.

[0082] In the method of this embodiment, by means of the multilingual processing capability of the model, the abstracts of the literature are translated into foreign language data such as English and Japanese data, respectively, so that the scientific research results and technical backgrounds in different language environments can be integrated into the model. Many international journals are often first published in English or other foreign languages. Translation can help the model access the knowledge in these foreign languages, make up for the information differences between different languages, expand the model's understanding of global scientific research trends, and improve the accuracy and comprehensiveness of label prediction in the field of technology; it can also improve multilingual processing capabilities. When GLM-4 is fine-tuning multilingual LoRA, it can better understand technical terms, concepts and expressions in different languages ​​by combining English and Japanese translation data. Differences between languages ​​(such as terminology usage, syntactic structure, etc.) may affect the interpretation of labels. The translated data can enable the model to learn the diversity of technical concepts in different languages, thereby improving the model's multi-dimensional understanding of label connotations, thereby improving the accuracy of label prediction.

[0083] In addition, the same technical concept or label in different languages ​​may be expressed in different ways. By translating, the description of the same label or technical field in different languages ​​is input into the model, which can help GLM-4 identify and unify technical labels in each language, so that it has a more consistent understanding of labels and technical fields in cross-language texts. This cross-language consistency is very important, especially when dealing with multilingual document repositories, to ensure the accurate transmission and application of label definitions. It can also enhance the accuracy of multilingual document recommendation and retrieval. In a multilingual environment, the retrieval and recommendation of scientific research documents often require processing text data from different languages. Through translation, the model can better understand the content of documents and their labels in different languages, and improve the accuracy of the document recommendation system. For example, through the translation data of English and Japanese, the model can help researchers establish associations between documents in different languages, so as to recommend more relevant technical documents. It can also improve the depth and semantic richness of label definitions. The definition of technical labels depends not only on the vocabulary of the language, but also on the language-specific context and cultural background. The difference in expression in different languages ​​may affect the depth and breadth of label definitions. Through multi-language processing, GLM-4 can obtain different expressions of technical labels in different languages ​​and the details behind them, thereby enhancing the in-depth understanding and accurate interpretation of label definitions. This multi-angle understanding helps to accurately analyze the true meaning behind technical labels.

[0084] In order to enhance the label prediction accuracy of the GLM-4 model when dealing with complex technical concepts, this paper proposes an improvement strategy: before label classification prediction training, the technical field and the labels to be classified are first input into the GLM-4 model to generate the definition of each label in the corresponding field, and the obtained label definition is input to GLM-4 as part of the input during LoRA fine-tuning and prediction. In this way, the model can more accurately understand the meaning of each label when fine-tuning LoRA, capture the part of the literature abstract related to the label definition content, accurately use the massive knowledge stored in the large language model for prediction, reduce the wrong prediction caused by concept confusion due to inaccurate label understanding, and thus improve the overall classification accuracy.

[0085] Specifically, for the key technology classification labels in the field of "humanoid robots" [artificial intelligence, motion control, environmental perception, human-computer interaction, mechanical design and manufacturing], the prompt "Please explain in detail the definition of the technologies in the field of humanoid robots [artificial intelligence, motion control, environmental perception, human-computer interaction, mechanical design and manufacturing]" is input into GLM-4, and GLM-4 is asked to output the definitions of these labels. The output of GLM-4 is as follows Figure 7 shown.

[0086] In order to fully utilize the multi-language processing capabilities of the large language model, this paper inputs Chinese literature abstract data into GLM-4 and asks GLM-4 to translate it into English and Japanese as data for LoRA fine-tuning. Specifically, "Please translate the abstract of the following literature text into English: x" is input into GLM-4 as a prompt, and GLM-4 outputs the translation result. The translation result is as follows Figure 8 shown.

[0087] After obtaining the translation data, the GLM-4 model is fine-tuned by LoRA based on the explanation of the key technical labels obtained above, using prompts such as Fig. 9 shown.

[0088] Among them, "explanation" represents the definition of key technical labels, and summary represents summary data of a language. Different languages ​​are divided into different training data to fine-tune the model. Using translation data for LoRA fine-tuning helps deepen the model's semantic understanding in different language systems. By finding the key semantics contained in specific syntactic structures, identifying vocabulary combinations with important meanings, and extracting the most representative and discriminative feature information, it helps the model classification.

[0089] After fine-tuning LoRA, the fine-tuned LoRA model is used to predict the validation set. The predictions are made for the data sets of three different languages ​​respectively, and the final answer is obtained by voting. The voting scheme is that when two or more languages ​​think that a label is correct, the label is added to the final answer. The process is as follows: Figure 3 As shown in the figure, voting and integrating the results obtained in different languages ​​can give full play to the advantages of multilingual information, avoid the limitations and deviations that may be brought about by a single language, obtain more accurate and reliable model classification results, and significantly improve the classification accuracy of the model.

[0090] experiment

[0091] Data preprocessing

[0092] The document data in the document classification data set of this article include but are not limited to papers, patent texts, etc. Papers include but are not limited to journal papers, master's and doctoral theses of universities, etc. This article uses a large language model and combines manual sorting to annotate the key technologies contained in the document abstracts. Specifically, this article uses chatglm, Wenxinyiyan and ChatGPT to annotate the technologies in the document abstracts. The first step is to determine which key technology tags need to be annotated, and ask the three large language models separately, "What key technologies does a include?", where a represents a future industry such as "brain-computer interface", then the whole sentence is "What key technologies does the brain-computer interface include?". The answer given by the large model is as follows: Fig.10 As shown in the table, it can be seen that the large language model has some "consensus" on the key technologies in the field of "brain-computer interface", but there are also some different opinions, or the content is the same but the words expressed are inconsistent. To address these problems, manual unified sorting is required to summarize the final classification label C.

[0093] After obtaining the classification labels, the abstract of the document text and the key technology labels are input into the large language model, and the document abstract is annotated with the help of the large language model. The prompt for inputting the large language model is "x, the above document clearly mentions Which technologies are in the literature? Where x is the abstract of the literature, is a set of key technology classification labels for the literature. For the above-mentioned question in the literature abstract “Based on stochastic resonance… brain-computer interface technology, first…”, the answers given by each large language model are as follows Fig.11 shown.

[0094] The above example is the most ideal answer given by the large language model. However, the answer given by the un-fine-tuned large language model is not always ideal. Sometimes the format changes, and sometimes the answer given is inconsistent with the key technical labels. Therefore, manual intervention is required to integrate the answers of the three large models, and based on their explanations and combined with manual summary and analysis, obtain multi-classification labels for the literature abstract.

[0095] Generally keep it in 2 rows. If the height is not enough, make appropriate adjustments.

[0096] Experimental setup

[0097] Experimental environment: This experiment is built on the Pytorch platform. The experimental platform is configured with an NVIDIA A40 graphics card and an operating system Ubuntu 18.04.5. The Pytorch version is 2.4.0 and the development language version is Python 3.10.

[0098] Experimental parameters: Each set of data of the experimental model was trained for 5 epochs, the model training batch was 1, the LoRA learning rate was set to 0.00001, and the LoRA rank was 16.

[0099] Comparative experiment

[0100] This article selects the following methods as baseline methods:

[0101] GLM4: GLM-4 (General Language Model 4) is a new generation of base preprocessing large language model launched by Zhipu AI. It belongs to the GPT family and is specifically used for natural language processing tasks. Based on previous generations of models, GLM-4 has added multi-language support, supporting 26 languages ​​including Japanese, Korean, and German, and has made many improvements and optimizations, which can demonstrate strong capabilities in text generation, understanding, and reasoning tasks.

[0102] GLM4+Transformer: This method uses GLM4 to embed the document abstract, and then uses transformer to capture the features of the text vector. Using the massive knowledge of the large language model to embed the document text can obtain a higher quality embedding than using the pre-trained model, while using the transformer's self-attention to capture the key content of the document text vector for classification.

[0103] Wenxinyiyan: Wenxinyiyan (Baidu ERNIE Bot) is a large-scale generative pre-trained language model developed by Baidu, based on Baidu's knowledge-enhanced semantic understanding technology (ERNIE: Enhanced Representation through Knowledge Integration). Wenxinyiyan has powerful capabilities in natural language understanding, generation, and reasoning by combining large-scale data pre-training with knowledge graph technology. It is a member of Baidu's "Wenxin Big Model" family and is designed for a variety of natural language processing tasks, especially generation, dialogue, search, and cross-modal applications.

[0104] ChatGPT3.5: ChatGPT-3.5 is a large-scale generative pre-trained language model developed by OpenAI, belonging to the generative pre-trained model (GPT) family. This model has been significantly improved and optimized based on GPT-3, aiming to provide higher-quality text generation and understanding capabilities for natural language processing tasks. ChatGPT-3.5 inherits the core technologies of the GPT series of models, including autoregressive generation based on the Transformer architecture and large-scale pre-training, but further enhances its performance in many aspects, especially in scenarios such as dialogue, reasoning, and multi-tasking.

[0105] Evaluation indicators

[0106] This paper uses Micro-accuracy (Micro-Acc) and Micro-F1 score to evaluate the experimental results. Micro-Acc is the overall average result of the precision of all labels. Micro-F1 is an indicator that comprehensively considers precision and recall. It is calculated by micro-averaging the precision and recall of all labels in multi-label classification. It balances the accuracy and completeness of the model when predicting positive samples. Micro-Acc and Micro-F1 score are commonly used evaluation indicators for multi-class classification tasks, especially for processing class imbalanced data. The formula is as follows.

[0107] (1)

[0108] (2)

[0109] The Micro-Precision and Micro-Recall formulas are as follows.

[0110] (3)

[0111] (4)

[0112] TP (True Positive) means that the true category is a positive example and the predicted category is a positive example; TN (True Negative) means that the true category is a negative example and the predicted category is a negative example; FP (False Positive) means that the true category is a negative example and the predicted category is a positive example; FN (False Negative) means that the true category is a positive example and the predicted category is a negative example.

[0113] Experimental results and analysis

[0114] The experimental results are as follows Fig.12 shown.

[0115] Among them, "(manual)" means that the labeling result is obtained by manually sorting the answers of the model. Because the unfine-tuned large language model processes the classification task, the generated answers are uncertain and inconsistent with the labels, so the answers need to be sorted manually to obtain a higher accuracy. If the answers of the large language model are not sorted, the accuracy is low, as shown in the results of "GLM4" in the table. It can be seen from the results in the table that the method in this paper has achieved better results than other models. Compared with the baseline model GLM4, the Acc of this method has increased by 37.43 percentage points and the F1 has increased by 14.54 percentage points. Compared with the manually sorted GLM4, the Acc of this method has increased by 14.67 percentage points and the F1 has increased by 6.94 percentage points. Compared with other manually sorted large language models, the prediction results are better. Moreover, because the method in this paper uses LoRA to fine-tune the large language model, the answers obtained are consistent with the standard labels without sorting, so it is more convenient than manually sorting the answers of the large language model.

[0116] The method in this paper is subjected to ablation experiments to verify the effectiveness of each part. The experimental results are as follows: Fig.13 shown.

[0117] As can be seen from the table, each part of the method proposed in this paper has played a positive role in the overall performance of the model, among which LoRA fine-tuning has the greatest improvement on the model performance. It can be seen that LoRA fine-tuning is an efficient and convenient method for using large language models to handle specific downstream tasks. In addition, the interpretation of professional terminology labels and multilingual fine-tuning also improve the performance of the model.

[0118] This paper proposes a method for multi-label classification of literature abstracts based on LoRA fine-tuning of GLM4. Compared with the mainstream pre-trained model in related technologies for multi-text classification, the use of fine-tuned large language models for multi-label classification can use less data to obtain better results compared to previous multi-label classification methods, saving data annotation costs. At the same time, this paper uses the interpretation of technical labels combined with the field to deepen the large language model's understanding of the field technical labels during classification, reducing the classification errors caused by "cognitive" bias. In addition, this paper also uses multilingual training data to fine-tune the model to fully utilize the multilingual processing capabilities of the large language model to understand the content of the literature text, and summarizes the final answer by voting, which improves the accuracy of classification.

[0119] refer to Figure 4 As shown, another embodiment of the present application provides a text multi-label classification device, comprising:

[0120] A label definition generation module, used to input the technical field and the labels to be classified into the large language model, so that the large language model generates the definition of each label under the corresponding technical field;

[0121] An annotation module is used to annotate the document data using a large language model to obtain annotated data;

[0122] A fine-tuning module, used to perform LoRA fine-tuning on the large language model using the labeled data to obtain a fine-tuned model, wherein the fine-tuned model is used for multi-label classification;

[0123] The classification module is used to perform multi-label classification on the text based on the definition of each label using the fine-tuned model.

[0124] Exemplarily, the large language model is fine-tuned using the annotated data to obtain a fine-tuned model, including:

[0125] Inputting Chinese document abstract data into the large language model, and translating the Chinese document abstract data into at least two foreign languages ​​through the large language model as data for LoRA fine-tuning;

[0126] Based on the explanation of the technical label in the label definition and the at least two foreign languages, the large language model is fine-tuned using LoRA to obtain a fine-tuned model.

[0127] Exemplarily, based on the definition of each label, the fine-tuned model is used to perform multi-label classification on the text, including:

[0128] Using the fine-tuned model to predict the text data, predict the Chinese document abstract data and the data sets of at least two foreign languages ​​respectively, and obtain the final answer by voting;

[0129] When two or more languages ​​in the voting plan consider a label to be correct, the label will be added to the final answer.

[0130] Exemplarily, combining the explanation of the technical label in the label definition and the at least two foreign languages, performing LoRA fine-tuning on the large language model includes:

[0131] constructing a prompt template in combination with the explanation of the technical label in the definition of the label and the at least two foreign languages;

[0132] The prompt template is input into the large language model for multilingual LoRA fine-tuning.

[0133] Exemplarily, multi-label classification of text includes:

[0134] For a given literature summary dataset and a label set , find the tags related to the literature summary dataset, where Describes the number of abstracts, where Indicates the number of tags, each document Both Relevance, indicating whether the tag is related to the document.

[0135] Exemplarily, the device further comprises:

[0136] The evaluation module is used to evaluate the classification results using Micro-accuracy and Micro-F1 score. Micro-accuracy is the overall average result of the precision of all labels. Micro-F1 score is an indicator that comprehensively considers precision and recall, and is calculated by micro-averaging the precision and recall of all labels in multi-label classification.

[0137] Exemplarily, the large language model includes a GLM-4 model.

[0138] The text multi-label classification device of the embodiment of the present application can use a fine-tuned large language model for multi-label classification. Compared with previous multi-label classification methods, it can use less data to obtain better results and save data annotation costs. The use of technical label explanations combined with the field can deepen the large language model's understanding of the field technical labels during classification and reduce classification errors caused by "cognitive" bias. It also uses multilingual training data to fine-tune the model to fully utilize the multilingual processing capabilities of the large language model to understand the content of the document text, and summarizes the final answer by voting, thereby improving the accuracy of classification.

[0139] Another embodiment of the present application provides an electronic device, including a processor and a memory; when the processor executes the running program stored in the memory, the text multi-label classification method described in any embodiment of the present application is implemented.

[0140] For example, Figure 6 As shown, Figure 6 A structural diagram of an electronic device provided as a specific example includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604. The memory 603 is used to store computer programs; the processor 601 is used to execute the program stored in the memory 603, thereby realizing the text multi-label classification method provided in any embodiment of the present application.

[0141] In the above electronic device, the communication bus may be a peripheral component interconnect standard bus or an extended industrial standard architecture bus. The communication bus may include an address bus, a data bus, a control bus, etc. The communication interface is used to realize data exchange between the electronic device and other devices.

[0142] The memory 603 may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. In addition, the memory 603 may also include other storage devices located at the remote end of the processor 601. The processor 601 may be a general-purpose processor, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other types of programmable logic devices, discrete gate circuits or transistor logic devices, discrete hardware components, etc.

[0143] Another embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the text multi-label classification method described in any embodiment of the present application is implemented. Figure 6 As shown, Figure 6 The computer-readable storage medium shown is a CD 20, on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the text multi-label classification method described in any embodiment of the present application can be implemented.

[0144] The computer-readable storage medium of this embodiment may be any type of medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates multiple media. Common storage media types include magnetic media (such as floppy disks, hard disks, and tapes), optical media (such as DVDs), or semiconductor media (such as solid-state drives (SSDs)).

[0145] In the above embodiments, all or part of the functions may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the functions may be implemented by one or more computer instructions contained in a computer-readable storage medium. When the computer loads and executes these computer program instructions, the processes or functions described in the embodiments of the present invention may be partially or completely executed.

[0146] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. For example, computer instructions may be transmitted from a website, computer, server, or data center to another computer, website, server, or data center.

[0147] It should be noted that:

[0148] In the embodiments of the present application, the terms "include", "comprise" and any other variants thereof are intended to indicate a non-exclusive inclusion relationship, meaning that when referring to including certain elements, the process, method, article or device not only includes these elements, but also includes other elements that are not explicitly listed, or includes inherent elements related to the process, method, article or device. In the absence of further limitation, the use of "includes a ..." to express an element does not exclude the possibility that there may be other identical elements in the process, method, article or device that includes the element.

[0149] Through the description of the above-mentioned embodiment methods, those skilled in the art can clearly understand that these methods can be implemented by software plus the necessary general hardware platform. Of course, it can also be implemented by pure hardware, but in many cases, the former is usually a better implementation method. Based on this understanding, the technical solution of the embodiment of the present application, or the technical contribution made by the present application, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD, etc.), and contains several instructions to enable an image display device (such as a mobile phone, computer, server, air conditioning equipment or network equipment, etc.) to execute the various methods described in the embodiment of the present application.

[0150] The above is a specific implementation of the embodiment of the present application, but does not limit the scope of protection of the present application. Any changes or alternatives that can be thought of by any technician familiar with the technical field within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be determined according to the content of the claims.

Claims

1. A text multi-label classification method, characterized in that: include: Input the technical field and the labels to be classified into the large language model, so that the large language model generates the definition of each label under the corresponding technical field; Use a large language model to annotate the document data to obtain annotated data; Using the labeled data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model, wherein the fine-tuned model is used for multi-label classification; Based on the definition of each label, the text is classified into multiple labels using the fine-tuned model.

2. The method according to claim 1, characterized in that The method of using the labeled data to perform LoRA fine-tuning on the large language model to obtain a fine-tuned model includes: Inputting Chinese document abstract data into the large language model, and translating the Chinese document abstract data into at least two foreign languages ​​through the large language model as data for LoRA fine-tuning; Based on the explanation of the technical label in the label definition and the at least two foreign languages, the large language model is fine-tuned using LoRA to obtain a fine-tuned model.

3. The method according to claim 2, characterized in that The method of performing multi-label classification on the text based on the definition of each label and using the fine-tuned model includes: Using the fine-tuned model to predict the text data, predict the Chinese document abstract data and the data sets of at least two foreign languages ​​respectively, and obtain the final answer by voting; When two or more languages ​​in the voting plan consider a label to be correct, the label will be added to the final answer.

4. The method according to claim 2, characterized in that: The method of combining the explanation of the technical tag in the definition of the tag and the at least two foreign languages ​​to perform LoRA fine-tuning on the large language model includes: constructing a prompt template based on the explanation of the technical tag in the definition of the tag and the at least two foreign languages; and inputting the prompt template into the large language model for multi-language LoRA fine-tuning.

5. The method according to claim 1, characterized in that: The multi-label classification of text includes: For a given literature summary dataset and a label set , find the tags related to the literature summary dataset, where Describes the number of abstracts, where Indicates the number of tags, each document Both Relevance, indicating whether the tag is related to the document.

6. The method according to claim 1, characterized in that The method also includes: using Micro-accuracy and Micro-F1 score to evaluate the classification result; Micro-accuracy is the result of overall averaging the precision rates of all labels; Micro-F1 score is an indicator that comprehensively considers the precision rate and the recall rate, and is obtained by micro-averaging the precision rates and the recall rates of all labels in the multi-label classification.

7. A text multi-label classification device, characterized in that: include: A label definition generation module, used to input the technical field and the labels to be classified into the large language model, so that the large language model generates the definition of each label under the corresponding technical field; An annotation module is used to annotate the document data using a large language model to obtain annotated data; A fine-tuning module, used to perform LoRA fine-tuning on the large language model using the labeled data to obtain a fine-tuned model, wherein the fine-tuned model is used for multi-label classification; The classification module is used to perform multi-label classification on the text based on the definition of each label using the fine-tuned model.

8. The method according to any one of claims 1 to 6, characterized in that The large language model includes a GLM-4 model; An electronic device, characterized in that: It comprises a processor and a memory; when the processor executes the running program stored in the memory, the text multi-label classification method according to any one of claims 1 to 8 is implemented; A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text multi-label classification method as described in any one of claims 1 to 7 is implemented.

Images