Text classification method and device, electronic equipment and readable medium

Abstract

The invention relates to a text classification method and device, electronic equipment and a readable medium. The method comprises the following steps: acquiring an input text, and classifying the input text through a text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, the initial classification result comprises N candidate tags, each candidate tag is used for representing the category to which the input text belongs, and N is a positive integer; adopting the input text and the N candidate tags to construct a prompt word text; evaluating each candidate tag according to the cue word text through a large language model to obtain a target confidence coefficient of each candidate tag; and based on the target confidence of each candidate tag, selecting the target candidate tag with the highest target confidence from the N candidate tags as a target classification result. According to the scheme provided by the invention, the actual demand of the user can be accurately identified in a complex language scene, so that the accuracy and reliability of text classification are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a text classification method and device, an electronic device and a readable medium. BACKGROUND

[0002] With the development of information technology, the generation and accumulation of text data are increasing. Text classification is a core task in natural language processing (NLP), and is widely used in intent recognition, information retrieval, automatic question answering systems and sentiment analysis, and can improve the efficiency of text processing.

[0003] In related technologies, text classification mainly relies on machine learning models, such as Naive Bayes, Support Vector Machine (SVM) and Random Forest. These methods classify by manually extracting text features (such as term frequency, TF-IDF (Term Frequency-Inverse Document Frequency)). Although they perform well in some simple tasks, their classification effect is difficult to meet the needs of practical applications in complex language scenarios. SUMMARY

[0004] To solve or partially solve the problems in related technologies, the present application provides a text classification method, device, electronic device and readable medium, which can accurately identify the actual needs of users in complex language scenarios, thereby effectively improving the accuracy and reliability of text classification.

[0005] The first aspect of the present application provides a text classification method, which comprises:

[0006] An input text is obtained, and a text classification model is called to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result comprises N candidate labels, each candidate label is used to represent the category to which the input text belongs, and N is a positive integer;

[0007] The input text and N candidate labels are used to construct a prompt word text;

[0008] A large language model is called to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label;

[0009] select, based on the target confidence of each of the candidate labels, a target candidate label with the highest target confidence from the N candidate labels as a target classification result.

[0010] In an embodiment, the initial classification result further comprises an initial confidence of each of the candidate labels; and before the step of constructing a prompt text by using the input text and the N candidate labels, the method further comprises:

[0011] select, based on the initial confidence of each of the candidate labels, a preferred candidate label with the highest initial confidence from the N candidate labels;

[0012] determine whether the initial confidence of the preferred candidate label is greater than or equal to a set threshold value;

[0013] if the initial confidence of the preferred candidate label is less than the set threshold value, determine to perform the step of constructing a prompt text by using the input text and the N candidate labels.

[0014] In an embodiment, the method further comprises:

[0015] if the initial confidence of the preferred candidate label is greater than or equal to the set threshold value, take the preferred candidate label as the target classification result.

[0016] In an embodiment, the step of constructing a prompt text by using the input text and the N candidate labels comprises:

[0017] determine a current business scenario according to the input text;

[0018] invoke a target template matched with the current business scenario;

[0019] splice the input text and the N candidate labels into the target template to obtain a prompt text.

[0020] In an embodiment, the step of evaluating each of the candidate labels by the large language model according to the prompt text to obtain a target confidence of each of the candidate labels comprises:

[0021] input the input text and the prompt text into the large language model;

[0022] evaluate, by the large language model, a matching degree of each of the candidate labels in the prompt text with the input text according to a deep semantic analysis of the input text and the prompt text, to obtain the target confidence of each of the candidate labels.

[0023] In an embodiment, the classifying the input text by the text classification model to obtain an initial classification result comprises:

[0024] performing word segmentation on the input text to obtain a plurality of subwords;

[0025] inputting the input text after word segmentation into the text classification model to insert a preset mark in a specific position of the input text by the text classification model;

[0026] converting each of the subwords into a subword embedding vector and converting the preset mark into a mark embedding vector by the text classification model;

[0027] encoding each of the subword embedding vectors by the text classification model to obtain an initial text representation of each of the subword embedding vectors;

[0028] interacting each of the initial text representations with the mark embedding vector by the text classification model to obtain a target mark embedding vector; the target mark embedding vector carries context information of the entire input text;

[0029] encoding the target mark embedding vector by the text classification model to obtain a target text representation of the target mark embedding vector;

[0030] classifying the input text based on the target text representation by the text classification model to obtain an initial classification result comprising N candidate labels and an initial confidence of each of the candidate labels.

[0031] In an embodiment, before the calling the text classification model, the method further comprises:

[0032] obtaining a pre-trained text classification model, a test sample set, and a labeled data set; the test sample set comprises a plurality of test sample texts, and the labeled data set comprises a true label labeled for each of the test sample texts;

[0033] classifying a plurality of the test sample texts by the text classification model respectively to obtain a sample classification result of each of the test sample texts; the sample classification result comprises a preferred sample label of the test sample text;

[0034] matching the preferred sample label belonging to the same test sample text with the true label to obtain a label matching result of each of the test sample texts;

[0035] According to a label matching result of each of the test sample texts, a classification evaluation index of the text classification model is calculated; the classification evaluation index includes an accuracy, a precision, and a recall rate;

[0036] It is determined whether the classification evaluation index is greater than or equal to a preset index.

[0037] If the classification evaluation index is greater than or equal to the preset index, it is determined to perform the step of invoking the text classification model.

[0038] The second aspect of the present application provides a text classification device, the device comprising:

[0039] A preliminary classification module is configured to obtain an input text, and invoke a text classification model to classify the input text by using the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result includes N candidate labels, each of the candidate labels is used to represent a category to which the input text belongs, and N is a positive integer;

[0040] A prompt word text construction module is configured to construct a prompt word text by using the input text and the N candidate labels.

[0041] A secondary aggregation module is configured to invoke a large language model to evaluate each of the candidate labels according to the prompt word text by using the large language model to obtain a target confidence of each of the candidate labels.

[0042] A target candidate label selection module is configured to select a target candidate label with the highest target confidence from the N candidate labels as a target classification result based on the target confidence of each of the candidate labels.

[0043] The third aspect of the present application provides an electronic device, comprising:

[0044] A processor; and

[0045] A memory having executable code stored thereon, when the executable code is executed by the processor, the processor executes the method as described above.

[0046] The fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon, when the executable code is executed by a processor of an electronic device, the processor executes the method as described above.

[0047] The technical solution provided by the present application can include the following beneficial effects:

[0048] The scheme provided in the application obtains input text, and calls a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, the initial classification result includes N candidate labels, each candidate label is used to represent a category to which the input text belongs, and N is a positive integer; an input text and the N candidate labels are used to construct a prompt word text; a large language model is called to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label; and based on the target confidence of each candidate label, a target candidate label with the highest target confidence is selected from the N candidate labels as a target classification result. The application classifies the input text by introducing the BERT model, the BERT model can output more accurate N candidate labels, and then the large language model is introduced to perform deep semantic analysis on the N candidate labels, the large language model can perform more accurate result evaluation in a complex language scene, in addition, the introduction of the prompt word text can effectively guide the large language model to focus on a key semantic point, so that the large language model better understands the classification task, thereby effectively improving the accuracy and reliability of text classification.

[0049] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the application. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and wherein:

[0051] Figure 1 is a flow diagram of a text classification method according to an embodiment of the application;

[0052] Figure 2 is another flow diagram of a text classification method according to an embodiment of the application;

[0053] Figure 3 is a training flow diagram of a BERT model according to an embodiment of the application;

[0054] Figure 4 is a text classification structure diagram based on a BERT model combined with a large language model according to an embodiment of the application;

[0055] Figure 5 is a structural diagram of a text classification device according to an embodiment of the application;

[0056] Figure 6 is a structural diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0057] Embodiments of the present application will be described in more detail with reference to the drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0058] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms "first", "second", "third", etc. can be used in this application to describe various information, these information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0060] The related art mainly relies on machine learning models for text classification, such as Naive Bayes, Support Vector Machine (SVM) and Random Forest, which classify by manually extracting text features (such as word frequency, TF-IDF), although they perform well in some simple tasks, but in complex language scenarios, their classification effect is difficult to meet the needs of practical applications.

[0061] To solve the above problems, the embodiments of the present application provide a text classification method, which preliminarily classifies the input text by introducing a BERT model, the BERT model can output more accurate N candidate labels, and then a large language model is introduced to perform deep semantic analysis on the N candidate labels, the large language model can perform more accurate result evaluation in complex language scenarios, in addition, by introducing prompt word text, the large language model can be effectively guided to focus on key semantic points, so that the large language model better understands the classification task, thereby effectively improving the accuracy and reliability of text classification.

[0062] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0063] Figure 1is a flowchart of a text classification method shown in an embodiment of the present application.

[0064] Referring to Figure 1 The text classification method of the present application comprises:

[0065] S110, obtaining an input text and calling a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result comprises N candidate labels, each candidate label is used to represent a category to which the input text belongs, and N is a positive integer.

[0066] In the embodiments of the present application, the text classification system can be applied to a variety of text classification tasks, especially complex intent recognition, multi-intent text processing or classification tasks requiring higher accuracy, so the specific application scenarios of the text classification system can include the following:

[0067] Intelligent customer service system: classify the text input by the user to identify the actual needs of the user, such as "order inquiry", "return application" and the like.

[0068] Government hotline intelligent agent assistance system: according to the content of the appeal of the citizens, judge the specific matter to which it belongs, so as to assist the intelligent agent of the government hotline to process, such as "medical service", "social security", "housing problem" and the like.

[0069] Legal document processing: classifying long legal texts and marking the content involved by different legal provisions.

[0070] Sentiment analysis and public opinion monitoring: sentiment classification of the appeal text of the citizens on the government platform, and in-depth analysis of public opinion tendency combined with large language model.

[0071] The text classification system can include a text classification model, which is a machine learning model used to assign text to one or more categories, for example, the text classification model can be a BERT (Bidirectional Encoder Representations from Transformers, pre-trained language representation model), or an improved version of the BERT model, such as RoBERTa (Robustly Optimized BERT Pretraining Approach, robustly optimized BERT pre-training approach), StructBERT (aiming to improve the BERT model by incorporating language structure) and the like.

[0072] The BERT model and its improved versions have strong text representation capabilities. Through pre-training on a large amount of unsupervised text and fine-tuning on specific tasks, the performance of text classification is greatly improved. Among them, the BERT model and its improved versions can be applied to text intent classification, text matter division, and semantic analysis classification task scenarios.

[0073] In actual application, the text classification system can obtain the input text input by the user, which can be any of express query information, citizen appeal information, and to-be-processed documents. Then the text classification system can call the text classification model (BERT model / RoBERTa model / StructBERT model) so as to classify the input text by using the text classification model, thereby obtaining an initial classification result, wherein the initial classification result can include N candidate labels, each candidate label can be used to represent the category to which the input text belongs, and N is a positive integer.

[0074] In an example, in a text intent classification task, the text classification system can determine the user's intent according to the text content input by the user. For example, the text classification system can be an intelligent customer service system, the user can input the input text “I want to query the express status” to the intelligent customer service system, the intelligent customer service system receives the input text, calls the text classification model (BERT model / RoBERTa model / StructBERT model), and then inputs the input text into the text classification model. The text classification model can perform embedding representation on the input text, and then use a fully connected neural network to classify the input text, thereby outputting N candidate labels, each candidate label being used to represent an intent category to which the input text belongs, such as “express query”, “payment problem”, “invoice problem”, and the like.

[0075] In another example, in a text matter division task, the text classification system can be used to divide a long text into several parts and assign a specific candidate label to each part. For example, in news classification or legal contract review, different paragraphs can involve different events or matters. The user can input a to-be-processed document (such as a legal document) to the text classification system, the text classification system receives the input text, calls the text classification model (BERT model / RoBERTa model / StructBERT model), and then inputs the input text into the text classification model. The text classification model can encode the input text sentence by sentence or paragraph by paragraph, and then use its classifier to classify each part, thereby outputting N candidate labels corresponding to each part for reference by the user or a downstream system.

[0076] S120, adopting the input text and the N candidate labels, constructing prompt word text.

[0077] Although the BERT model and its variant models (e.g., the RoBERTa model, the StructBERT model, etc.) perform well in the text classification task, their classification accuracy depends on the pre-training effect and the richness of the labeled data set, and thus may face the problem of inaccurate initial classification results. For example, when facing complex or ambiguous input texts, the BERT model and its variant models still have some key problems:

[0078] Limited classification accuracy: The initial classification result output by the text classification model (e.g., the BERT model) cannot fully reflect the true intention in a complex context due to the dependence on the pre-defined classifier structure. In particular, in a multi-classification task, although the text classification model can output multiple candidate labels, it is difficult to automatically select the most suitable candidate label.

[0079] Poor handling of ambiguous texts: The initial classification result output by the text classification model is often not accurate when processing semantically ambiguous or multi-intention texts. For example, when the user inputs the input text "I want to return the goods and query the express status", the text classification model may not be able to correctly identify both the "return" and "express query" intention categories.

[0080] Lack of precise focus: If only the initial classification result output by the text classification model is relied on, the text classification system cannot further optimize the initial classification result after obtaining multiple candidate labels. Although the model parameters of the text classification model can be manually adjusted or the most suitable candidate label can be selected through manual intervention, this will increase the manual cost and lack an automated secondary optimization mechanism.

[0081] To this end, the embodiments of the present application can further optimize the initial classification result based on the initial classification result output by the text classification model by introducing a prompt word text and a large language model.

[0082] Specifically, the text classification system can also include a prompt word extraction system and a large language model. Therefore, after obtaining the N candidate labels output by the text classification model, the text classification system can first call the prompt word extraction system to construct a prompt word text (Prompt) by using the input text and the N candidate labels through the prompt word extraction system.

[0083] S130, calling a large language model to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label.

[0084] After obtaining the prompt word text output by the prompt word extraction system, the text classification system can call a large language model (LLM, Large Language Model) again. The large language model is a deep learning model with strong natural language understanding and generation capabilities. Specifically, the large language model can be any one of the following: general-purpose Q&A model, ChatGLM (Chat Generative Language Model), and Baichuan (a large language model with over 100 billion parameters developed by Baichuan Intelligence).

[0085] Since the prompt word text can effectively guide the optimization output of the large language model, the large language model can evaluate each candidate label based on the prompt word text to output the target confidence of each candidate label.

[0086] S140, based on the target confidence of each candidate label, selects a target candidate label with the highest target confidence from the N candidate labels as a target classification result.

[0087] The target confidence (Confidence) is the degree of confidence of the large language model for each candidate label, which can be calculated by a Softmax (normalized exponential) function.

[0088] After obtaining the target confidence of each candidate label output by the large language model, the text classification system can select a candidate label with the highest target confidence from the N candidate labels based on the target confidence of each candidate label, and then output the target candidate label as the target classification result.

[0089] It can be seen that the embodiments of the present application can accurately identify the actual needs of the user in a complex language scenario by combining the advantages of the text classification model (such as the BERT model) and the large language model, thereby effectively improving the accuracy and reliability of text classification. Specifically, the BERT model performs well in text embedding and semantic understanding, and can well capture the complex semantic relationship in the input text. Therefore, by introducing the BERT model to preliminarily classify the input text, the BERT model can output more accurate N candidate labels, which can provide a reliable basis for subsequent secondary focusing optimization. The large language model (such as ChatGLM, Tongyiqianwen, and other GPT (Generative Pre-trained Transformer) type large language models) can evaluate the rationality of each candidate label from the aspects of semantic depth and context relevance, so that the final target classification result is not only based on the surface similarity of the candidate label, but also considers the deep semantic consistency between the candidate label and the input text. Therefore, by introducing the large language model to perform secondary focusing on the N candidate labels output by the BERT model, higher accuracy and stability can be achieved when processing complex text or ambiguous text or multi-intent text, and by introducing the prompt word text, the large language model can be effectively focused on the key semantic point, thereby further improving the accuracy and reliability of text classification.

[0090] As can be seen from the example, the scheme provided by the present application acquires the input text and calls a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result includes N candidate labels, each candidate label is used to represent the category to which the input text belongs, and N is a positive integer; the input text and the N candidate labels are used to construct a prompt word text; a large language model is called to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label; and based on the target confidence of each candidate label, a target candidate label with the highest target confidence is selected from the N candidate labels as a target classification result. The present application introduces the BERT model to preliminarily classify the input text, and the BERT model can output more accurate N candidate labels, and then introduces the large language model to perform deep semantic analysis on the N candidate labels. The large language model can perform more accurate result evaluation in a complex language scenario, and in addition, by introducing the prompt word text, the large language model can be effectively focused on the key semantic point, so that the large language model better understands the classification task, thereby effectively improving the accuracy and reliability of text classification.

[0091] Figure 2 is another flowchart of the text classification method shown by the present application.

[0092] Referring toFigure 2 The text classification method of the present application comprises:

[0093] S210, obtaining an input text, and calling a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result comprises N candidate labels and initial confidence of each candidate label, each candidate label is used to represent a category to which the input text belongs, and N is a positive integer.

[0094] The text classification system can be applied to various text classification tasks, especially complex intent recognition, multi-intent text processing or classification tasks requiring higher accuracy.

[0095] The text classification system can include a text classification model, which can be any one of a BERT model, a RoBERTa model and a StructBERT model, wherein the RoBERTa model and the StructBERT model are both variant models of the BERT model (i.e. improved versions of the BERT model).

[0096] The BERT model and its variant models have strong text representation capabilities. Such models are pre-trained on large-scale unsupervised text and then fine-tuned on specific tasks, greatly improving the performance of text classification.

[0097] In actual application, the text classification system can obtain an input text input by a user, which can be any one of express query information, citizen appeal information, and a to-be-processed document. Then the text classification system can call a text classification model (BERT model / RoBERTa model / StructBERT model) to classify the input text using the text classification model, thereby obtaining an initial classification result, wherein the initial classification result can include N candidate labels and initial confidence of each candidate label, each candidate label can be used to represent a category to which the input text belongs, N is a positive integer, and the initial confidence (Confidence) is the degree of confidence of the text classification model for each candidate label, which can be calculated by a Softmax function.

[0098] In an embodiment, before calling the text classification model, the method can further comprise:

[0099] obtaining a pre-trained text classification model, and obtaining a test sample set and a labeled data set; the test sample set includes a plurality of test sample texts, and the labeled data set includes real labels labeled for each test sample text; inputting the plurality of test sample texts into the text classification model respectively to classify the plurality of test sample texts respectively by the text classification model, and obtaining sample classification results of each test sample text; the sample classification result includes an optimal sample label of the test sample text; matching the optimal sample label belonging to the same test sample text with the real label to obtain a label matching result of each test sample text; calculating a classification evaluation index of the text classification model according to the label matching result of each test sample text; the classification evaluation index includes accuracy, precision and recall; judging whether the classification evaluation index is greater than or equal to a preset index; if the classification evaluation index is greater than or equal to the preset index, it is determined that the step of calling the text classification model is executed.

[0100] Before calling the text classification model, the performance of the text classification model can be evaluated, so as to determine whether to call the text classification model based on the performance of the text classification model.

[0101] Specifically, taking the text classification model as the BERT model for example, Figure 3 is a training flowchart of the BERT model, and the training process of the BERT model mainly includes a pre-training process and a fine-tuning process, Figure 3Specifically, the fine-tuning process, wherein the pre-training process can use an unsupervised learning algorithm to train the BERT model on a large amount of text, so that it learns the language patterns and structures in the text, and then saves the pre-trained BERT model (BERT pretrained model) for entering the fine-tuning process: for a specific text classification task, prepare the corresponding training sample set, which can include training sample texts to be classified and corresponding labels, input each training sample text into the tokenizer for tokenization processing to obtain multiple token type ids, then load the previously saved BERT pre-training model (BERT pretrained model), that is, load the previously pre-trained BERT model. In order to make the BERT pre-training model better learn the deep semantic knowledge and semantic steps of the language, the embodiments of the present application can mask some tokens in each training sample text to obtain an attention mask, so as to fine-tune the pre-training model (BERT pretrained model) using the training sample text, the corresponding label, the token type id and the attention mask, that is, to fine-tune the pre-training model (BERT pretrained model) using a supervised learning algorithm, so that it can predict the correct label according to the text content.

[0102] The pre-trained text classification model is obtained in the above manner, that is, the fine-tuned BERT model (New model) is obtained, at this time the test sample set and the labeled data set can be obtained to test and evaluate the fine-tuned BERT model using the test sample set and the labeled data set.

[0103] In a specific implementation, the test sample set can include a plurality of test sample texts, the labeled data set can include a ground truth labeled by a test personnel for each test sample text, the text classification system can input the plurality of test sample texts into the fine-tuned BERT model respectively, so that the BERT model classifies each test sample text, and then outputs a sample classification result of each test sample text, wherein the sample classification result can include a preferred sample label of the test sample text, the preferred sample label is used to represent a category to which the test sample text belongs, the text classification system can match the preferred sample label belonging to the same test sample text with the ground truth, so as to obtain a label matching result of each test sample text, the label matching result can include any one of TP (True Positives), FP (False Positives), FN (False Negatives) and TN (True Negatives), wherein TP refers to that the BERT model predicts that a certain test sample text is a positive sample and actually it is also a positive sample, FP refers to that the BERT model predicts that a certain test sample text is a positive sample, but actually it is a negative sample, FN refers to that the BERT model predicts that a certain test sample text is a negative sample, but actually it is a positive sample, and TN refers to that the BERT model predicts that a certain test sample text is a negative sample and actually it is also a negative sample. The text classification system can calculate a classification evaluation index of the BERT model according to the label matching result of each test sample text, wherein the classification evaluation index can include accuracy, precision and recall.

[0104] Accuracy: the proportion of correctly classified samples to the total number of samples:

[0105]

[0106] wherein the total number of samples refers to the total number of test sample texts, the number of correctly classified samples refers to the number of the predicted results (preferred sample labels) of the BERT model consistent with the actual results (ground truths) of artificial labeling, for example, the number of correctly classified samples can be the sum of TP and TN.

[0107] Precision: the proportion of true positive samples to the samples predicted by the BERT model as positive samples:

[0108]

[0109] wherein TP is a true positive sample and FP is a false positive sample.

[0110] Recall: the proportion of positive samples in the true positive class predicted by the BERT model:

[0111]

[0112] where FN is the false negative class sample.

[0113] F1 Score: the harmonic mean of precision and recall:

[0114]

[0115] The above classification evaluation indicators can quantify the performance of the BERT model in different classification tasks to evaluate the performance of the BERT model.

[0116] In a specific implementation, the text classification system can determine whether the classification evaluation indicators of the BERT model are greater than or equal to the preset indicators, wherein the preset indicators can include a preset accuracy and a preset score, so the text classification system can determine whether the accuracy of the BERT model is greater than or equal to the preset accuracy, and determine whether the F1 score of the BERT model is greater than or equal to the preset score. If the accuracy of the BERT model is ≥ the preset accuracy, and the F1 score of the BERT model is ≥ the preset score, the text classification system can evaluate that the performance of the BERT model is good, and then the BERT model can be called for preliminary classification, that is, it is determined to execute step S210. If the accuracy of the BERT model is < the preset accuracy, and / or the F1 score of the BERT model is < the preset score, the text classification system can evaluate that the performance of the BERT model is poor, and then the BERT model can be replaced by a model with better performance and more parameters, for example, the RoBERTa model or the StructBERT model. In addition, the training sample set of the BERT model can be increased to prevent overfitting of the BERT model, help the BERT model to better generalize, and improve the performance.

[0117] In an embodiment, the initial classification result obtained by classifying the input text by the text classification model can include:

[0118] The input text is segmented to obtain a plurality of subwords; the segmented input text is input into a text classification model to insert a preset mark in a specific position of the input text through the text classification model; each subword is converted into a subword embedding vector and the preset mark is converted into a mark embedding vector through the text classification model; each subword embedding vector is encoded through the text classification model to obtain an initial text representation of each subword embedding vector; the mark embedding vector is interacted with each initial text representation through the text classification model to obtain a target mark embedding vector; the target mark embedding vector carries context information of the entire input text; the target mark embedding vector is encoded through the text classification model to obtain a target text representation of the target mark embedding vector; the input text is classified based on the target text representation through the text classification model to obtain an initial classification result containing N candidate labels and an initial confidence of each candidate label.

[0119] The embodiments of the present application can use a pre-training model based on the Transformer (deep learning model) architecture such as BERT or RoBERTa or StructBERT for text classification tasks. For each input text, the text classification system can embed it into the input layer of the BERT model, and after the encoding processing of multiple Transformer layers, the text representation h i is obtained, where i represents the i-th subword of the input text, i is a positive integer.

[0120] Taking the BERT model as the text classification model and a single input text as an example: as shown in Figure 4 , the text content is the input text input by the user to the text classification system, and the text classification system can first preprocess the input text, for example, segment the input text into a subword sequence (or word sequence): {x1, x2, …, xn}. Where the subword sequence includes a plurality of subwords, n is a positive integer, i≤n, i is the index of a certain position in the input text, and n is the index of the end position in the input text, for example, x n is the i-th subword in the input text, and x i is the last subword in the input text. nis the nth subword in the input text, or in some tasks, it can represent a specific input instance. Among them, the subword refers to the smallest unit carrying semantic information of the text. For example, assuming that the input text is “I want to return goods and query the express status”, the input text will be segmented into n subwords: “I”, “want”, “to return goods”, “and”, “query”, “express”, “status”. The text classification system can input the segmented input text into the BERT classifier, which is a BERT model (text classification model). The BERT model can insert a preset mark [CLS] (Classification Language Substitution) in a specific position of the input text. Specifically, in the BERT model, [CLS] is a special mark representing the overall information of the input text, which is used for downstream classification tasks. It is usually inserted by the BERT model in the first position of the input text, so the specific position of the input text can specifically refer to the first position of the input text, thereby obtaining: {[CLS], I, want, to return goods, and, query, express, status}.

[0121] After that, each subword (or word) is converted into a subword embedding vector (or word embedding vector) X i :

[0122] X i = Embedding(x i )

[0123] Among them, the Embedding function maps the subword to a high-dimensional space to obtain the subword embedding vector X i .

[0124] Similarly, the preset mark [CLS] can also be converted into a mark embedding vector by the word embedding layer of the BERT model.

[0125] After that, each subword embedding vector X i is encoded by multiple Transformer layers of the BERT model (such as self-attention mechanism and feedforward neural network processing), and all subword embedding vectors can be influenced and updated based on their context, thereby obtaining the context representation h i :

[0126] h i = Transformer(x i )

[0127] Among them, h i is the subword embedding vector X iThe initial text representation.

[0128] Then, the BERT model can aggregate the label embedding vectors [CLS] of the entire input text to form the final text representation h of the input text. CLS In its implementation, the BERT model can use a self-attention mechanism to associate the label embedding vector [CLS] with each initial text representation h. i Interacting with the target text allows the label embedding vector [CLS] to carry the contextual information of the entire input text. For ease of distinction, the interacted label embedding vector [CLS] can be called the target label embedding vector [CLS], which carries the contextual information of the entire input text. Similarly, the target label embedding vector [CLS] can also be encoded by the Transformer layer of the BERT model to obtain the target text representation h of the target label embedding vector [CLS]. CLS Target text representation h CLS It incorporates the contextual information of the entire input text, i.e., the target text representation h. CLS It can contain all the initial text representations h from the input text. i Information.

[0129] Subsequently, the target text representation is obtained through h CLS The classification result y is obtained after passing through the fully connected layer and the softmax layer of the BERT model:

[0130] y = softmax(W·h) CLS +b)

[0131] Where W represents the learnable weight matrix, b represents the bias term, and the classification result can include M candidate labels and the initial confidence scores corresponding to the M candidate labels, where M is a positive integer.

[0132] The initial confidence level represents the degree of confidence the BERT model has in each candidate label. It can be calculated using the Softmax function, which transforms the raw output of the classifier in the BERT model into a probability distribution. The formula for calculating the initial confidence level is as follows:

[0133]

[0134] Among them, z i is the output score of the BERT model for the i-th category, and N is the total number of categories.

[0135] The BERT model can be based on the initial confidence score p(y) of each candidate label. i|x), sort the M candidate labels in descending order, and then select the top N candidate labels {y1,y2,…,y} from the M candidate labels. N}, where N is a positive integer and N≤M, the BERT model can assign N candidate labels and their corresponding initial confidence scores P(y) to each candidate label. i The output of |x) is the initial classification result.

[0136] S220, based on the initial confidence of each candidate label, select the preferred candidate label with the highest initial confidence from the N candidate labels.

[0137] To ensure a rapid response to simple requests and accurate processing of complex requests, embodiments of this application can compare the initial confidence levels P(y) of each candidate label. i |x), determine the category to which the BERT model most likely predicts the input text, and then determine whether to enter the secondary aggregation analysis stage of the large language model based on the category.

[0138] Secondary focusing (SF) refers to further analyzing and optimizing the initial classification results by incorporating additional information (such as cue text) to improve the accuracy and reliability of the final classification.

[0139] like Figure 4 As shown, the text classification system can be based on the initial confidence score P(y) of each candidate label. i From candidate label 1 to candidate label N, select the candidate label with the highest initial confidence as the preferred candidate label (Top-1 label). Then, the preferred candidate label (Top-1 label) enters the confidence threshold determination step: If the initial confidence of the preferred candidate label (Top-1 label) is high enough, the text classification system can directly return the preferred candidate label (Top-1 label) to the user, that is, there is no need to enter the secondary aggregation analysis stage of the large language model, thereby reducing subsequent processing and ensuring a rapid response to simple requests; If the initial confidence of the preferred candidate label (Top-1 label) is not high, the text classification system can enter the secondary aggregation analysis stage of the large language model to use the large language model to perform secondary aggregation analysis on the initial classification results, thereby improving classification accuracy and ensuring accurate processing of complex requests.

[0140] S230, determine whether the initial confidence level of the preferred candidate label is greater than or equal to the set threshold.

[0141] The embodiment of the application can pre-configure a set threshold τ, and then the text classification system can determine whether the initial confidence P(y1|x) of the preferred candidate label (Top-1 label) is greater than or equal to the set threshold τ. If P(y1|x)≥τ, go to step S240; if P(y1|x)<τ, go to step S250.

[0142] S240, if the initial confidence of the preferred candidate label is greater than or equal to the set threshold, the preferred candidate label is taken as the target classification result.

[0143] As shown in Figure 4 if P(y1|x)≥τ, the text classification system can directly output the selected candidate label (Top-1 label) as the target classification result, that is, directly return the preferred candidate label (Top-1 label) to the user:

[0144] S250, if the initial confidence of the preferred candidate label is less than the set threshold, the input text and N candidate labels are used to construct a prompt word text.

[0145] Although the BERT model and its variant models (such as the RoBERTa model, the StructBERT model, etc.) perform well in the text classification task, their classification accuracy depends on their pre-training effect and the richness of the labeled data set, and thus they may face the problem of inaccurate initial classification results. For example, when facing complex or ambiguous input texts, the BERT model and its variant models still have some problems. In this regard, the embodiment of the application can effectively solve the problems of the BERT model and its variant models by introducing the prompt word text and the large language model, for example:

[0146] Insufficient processing of multi-intent text: the text classification model (BERT model / RoBERTa model / StructBERT model) may not be ideal in classification when facing complex and ambiguous input texts. In this regard, the embodiment of the application can more accurately identify and classify multi-intent texts by introducing the prompt word text and the large language model.

[0147] Low accuracy of classification results: if the text classification system only relies on the initial classification results (candidate label 1-candidate label N) output by the text classification model (BERT model / RoBERTa model / StructBERT model), it lacks effective subsequent optimization mechanism. In this regard, the embodiment of the application uses the large language model to focus on the initial classification results (candidate label 1-candidate label N) again, which can dynamically evaluate the candidate labels and automatically select the most accurate category, thereby improving the reliability of the classification results.

[0148] The text classification model is less adaptable: the text classification model (BERT model / RoBERTa model / StructBERT model) has limited performance in different application scenarios. To this end, the embodiments of the present application combine the text classification model with the large language model, and the secondary focus analysis link of the large language model can provide a flexible adaptation mechanism for the text classification system, so that the text classification system can be optimized for different fields and tasks.

[0149] Therefore, the embodiments of the present application can effectively improve the accuracy of text classification and the ability to process complex text by combining prompt word text and large language model for secondary focus on the N candidate labels output by the text classification model, and have wide application prospects and technical value.

[0150] In a specific implementation, the text classification system can also include a prompt word extraction system and a large language model, so that after determining that the initial confidence P(y1|x) of the preferred candidate label (Top-1 label) is less than the set threshold τ, the text classification system can first call the prompt word extraction system to construct the prompt word text by using the input text and the N candidate labels.

[0151] In an example, as shown in Figure 4 If P(y1|x)<τ, the prompt word extraction system constructs the classification prompt word text (i.e., prompt word text) using the text content (i.e., input text) and Top-N candidate labels (i.e., N candidate labels): "You are an intelligent customer service assistant, you have the ability to judge user intent, please judge the most consistent user's real intent from N classification labels according to the user input text, and output the most consistent label", and then the text classification system can input the text content (i.e., input text) and the classification prompt word text (i.e., prompt word text) into the large language model (Large Language Model) together, and the large language model further processes the labels 1- N (i.e., N candidate labels) output by the BERT classifier (i.e., text classification model).

[0152] In an embodiment, constructing the prompt word text using the input text and the N candidate labels can include:

[0153] Determining the current business scenario according to the input text; calling a target template matched with the current business scenario; concatenating the input text and the N candidate labels into the target template to obtain the prompt word text.

[0154] The embodiments of the present application can pre-configure multiple business scenarios and templates corresponding to each business scenario, and the multiple business scenarios can include the following:

[0155] Intelligent customer service system: classify the user input text, identify the user's actual needs, such as "order inquiry", "return application" and other needs.

[0156] Intelligent hot line assistant system: according to the citizen's appeal content, judge the specific matter it belongs to, and assist the intelligent hot line to process, such as "medical service", "social security", "housing problem" and other matters.

[0157] Legal document processing: classify long legal texts and mark the content involved in different legal provisions.

[0158] Sentiment analysis and public opinion monitoring: sentiment classification of citizen appeal text on the government platform, and in-depth analysis of public opinion tendency combined with large language model.

[0159] The prompt word extraction system can determine the current business scenario according to the input text, and then call the target template matching the current business scenario from the pre-configured multiple templates, and then splice the input text and N candidate labels into the target template, so as to construct the prompt word text.

[0160] In an example, assuming the input text is "I have an old person at home who needs care, can I apply for low-income or social assistance?", it can be determined that the input text is the citizen's appeal information, so it can be determined that the current business scenario is the business scenario based on the intelligent hot line assistant system, and then the target template matching the current business scenario can be called, and then the input text and N candidate labels (such as "medical service", "social security" and "housing problem" candidate labels) are spliced into the target template, so as to construct the prompt word text: "You are a government service expert, please judge the most suitable candidate label according to the following citizen's appeal content. The candidate labels are: medical service, social security, housing problem".

[0161] S260, calling a large language model to evaluate each candidate label according to the prompt word text through the large language model to obtain the target confidence of each candidate label.

[0162] After obtaining the prompt word text output by the prompt word extraction system, the text classification system can call the large language model again, which is a deep learning model with strong natural language understanding and generation capabilities. Among them, the large language model can be any one of the general thousand questions model, ChatGLM, Baichuan.

[0163] Because the prompt word text can effectively guide the optimization output of the large language model, the large language model can evaluate each candidate label according to the prompt word text, so as to output the target confidence of each candidate label.

[0164] In an embodiment, the target confidence of each candidate label is obtained by evaluating each candidate label according to the prompt word text by the large language model, which can include:

[0165] The input text and the prompt word text are input into the large language model, and the matching degree of each candidate label in the prompt word text with the input text is evaluated according to the deep semantic analysis of the input text and the prompt word text by the large language model, so as to obtain the target confidence of each candidate label.

[0166] The text classification system can input the input text and the prompt word text into the large language model, so that the large language model performs deep semantic analysis on the input text and the prompt word text, and then evaluates the matching degree of each candidate label in the prompt word text with the input text according to the deep semantic analysis of the input text and the prompt word text, so as to calculate the target confidence of each candidate label according to the matching degree:

[0167]

[0168] Wherein, s i is the matching score of the i-th candidate label evaluated by the large language model based on the prompt word text p, and the matching score is used to represent the matching degree of the i-th candidate label with the input text.

[0169] In an example, assuming that the input text is "I have an old person to take care of at home, can I apply for low-income or social assistance?", the corresponding constructed prompt word text is "You are a government service expert, please judge the most suitable candidate label according to the following citizen's appeal content. The candidate labels are: medical service, social security, housing problem", then the large language model performs deep semantic analysis on the input text and the prompt word text, and evaluates the matching degree of each candidate label (such as "medical service, social security, housing problem") in the prompt word text with the input text according to the deep semantic analysis of the input text and the prompt word text, so as to calculate the target confidence of each candidate label according to the matching degree, for example, the target confidence of "social security" is 0.85, the target confidence of "housing problem" is 0.10, and the target confidence of "medical service" is 0.05.

[0170] S270, based on the target confidence of each candidate label, selecting a target candidate label with the highest target confidence from the N candidate labels as a target classification result.

[0171] After obtaining the target confidence of each candidate label output by the large language model, the text classification system can select a candidate label with the highest target confidence from the N candidate labels as a target candidate label based on the target confidence of each candidate label:

[0172]

[0173] wherein, is the target candidate label selected by the large language model after re-evaluation, P'(y i is the target confidence of the large language model on the candidate label y i .

[0174] The text classification system can output the target candidate label as the target classification result.

[0175] As can be seen from the example, the scheme provided by the present application has the following beneficial effects:

[0176] 1. Secondary focusing classification mechanism

[0177] Technical points: By using a large language model (such as ChatGLM, Tongyi Qianwen, etc. GPT large language model) to perform secondary focusing on the initial classification result output by the BERT model, the accuracy of the initial classification result is further optimized in combination with the prompt text, so as to obtain a target classification result with higher precision.

[0178] Advantages: BERT model can usually only output multiple candidate labels, and if the text classification system only relies on the initial classification result output by the BERT model, it lacks an automatic evaluation mechanism to select the most suitable category. The embodiment of the present application introduces a large language model to perform deep semantic analysis on N candidate labels, so that the text classification system can perform more accurate result evaluation in a multi-classification scenario with the help of the large language model, especially when processing complex texts or multi-intent texts, it shows higher accuracy and stability.

[0179] 2. Optimization mechanism guided by prompt text

[0180] Technical points: Use the prompt text (Prompt) mechanism to combine the input text and the prompt text into the large language model during the secondary focusing process to improve the explainability and accuracy of the classification result.

[0181] Advantages: The prompt text can effectively guide the large language model to focus on key semantic points, so that it can better understand the classification task. Compared with direct classification in related technologies, the embodiment of the present application further narrows the classification range through the prompt text, which can improve the classification effect, especially when facing ambiguous and cross-domain input texts. In addition, the addition of the prompt text enables the large language model to dynamically adjust the classification standard according to the context, thereby adapting to the classification needs of different fields.

[0182] 3. Preliminary classification based on BERT model

[0183] Technical points: use BERT model or RoBERTa model or StructBERT model as a text classification model to preliminarily classify the input text, which can obtain more accurate N candidate labels, and provide a basis for the optimization of subsequent large language models.

[0184] Advantages: Compared with the text classification based on machine learning models (such as SVM or Naive Bayes) in related technologies, the BERT model performs better in text embedding and semantic understanding, and can better capture complex semantic relationships in input text. Therefore, by using the BERT model as the execution subject of the preliminary classification step, the embodiment of the present application can provide more accurate N candidate labels, and provide a reliable basis for subsequent secondary focus optimization.

[0185] 4. Dynamic evaluation mechanism

[0186] Technical points: dynamically evaluate each candidate label by a large language model, and determine the most suitable target candidate label as the final target classification result according to the target confidence ranking.

[0187] Advantages: Compared with the classification only based on simple score ranking in related technologies, the large language model used in the embodiment of the present application can evaluate the rationality of each candidate label from two aspects of semantic depth and context relevance. The dynamic evaluation mechanism provided by the embodiment of the present application can make the final target classification result not only based on the surface similarity of candidate labels, but also consider the deep semantic consistency between candidate labels and input text, thereby greatly improving the classification accuracy.

[0188] In order for those skilled in the art to better understand the embodiments of the present application, the following two examples are used to illustrate the embodiments of the present application.

[0189] Example one: the business scenario is text intent classification in an intelligent customer service system, and the specific process is as follows:

[0190] Step 1: the input text is preliminarily classified by the BERT-based text classification model.

[0191] Specific steps: input the user's input text (such as "I want to query the order status") into the BERT model, so that the BERT model outputs the first N most possible candidate labels (such as "order query", "payment problem", "invoice problem", etc.).

[0192] Result: N candidate labels are obtained, and the subsequent steps will be further optimized based on these candidate labels. Among them, in the current business scenario, each candidate label is used to represent the intent category to which the input text belongs.

[0193] Step 2: confidence threshold determination.

[0194] Specific steps: Determine the initial confidence of the Top-1 label output in the first step. If P(y1|x) ≥ τ (where P(y1|x) is the initial confidence of the Top-1 label, and τ is the set threshold), directly take the Top-1 label as the final target classification result.

[0195] Results: If the initial confidence of the Top-1 label is high enough, the text classification system can directly return the Top-1 label, thereby reducing subsequent processing and ensuring quick response to simple appeals; if the initial confidence of the Top-1 label is lower than the set threshold, proceed to the next step of analysis, thereby ensuring accurate processing of complex appeals.

[0196] Step 3: Combine the demand description to build a prompt word text related to the candidate label.

[0197] Specific steps: According to the input text and N candidate labels, build a prompt word text: "You are an intelligent customer service assistant, you have the ability to judge user intent, please judge the following N candidate labels according to the user input text, and output the most suitable candidate label. Candidate labels include: order query, payment problem, invoice problem."

[0198] Results: Get the prompt word text, combine the input text with the prompt word text for subsequent large language model analysis.

[0199] Step 4: Perform secondary focus analysis on N candidate labels through a large language model (such as ChatGLM model) to evaluate the accuracy of each candidate label.

[0200] Specific steps: Input the prompt word text and the original input text into the ChatGLM model to perform deep semantic analysis and classification optimization on the N candidate labels using the ChatGLM model, and generate evaluation results for each candidate label (for example, the target confidence of "order query" is 0.85, the target confidence of "payment problem" is 0.10, and the target confidence of "invoice problem" is 0.05).

[0201] Results: Get the evaluation results for each candidate label, and the ChatGLM model outputs the most suitable target candidate label (for example, "order query").

[0202] It can be seen that, by combining the prompt word text and the large language model, the accuracy of complex text classification can be effectively improved. For example, in the intelligent customer service scenario, the user's input intention may be ambiguous, but through the deep semantic analysis of the large language model, the text classification system can more accurately identify the user's actual needs, thereby optimizing the business process, reducing misclassification, achieving automated optimization of the classification task, reducing the need for manual intervention, and improving the intelligent level of the intelligent customer service scenario.

[0203] Example two: the business scenario is the text matter division in the intelligent agent auxiliary system of the government hotline, and the specific process is as follows:

[0204] Step 1: the text classification model based on RoBERTa performs preliminary classification on the citizen's appeal text.

[0205] Specific steps: the user inputs the input text containing the appeal content, and the RoBERTa model outputs the top N most possible candidate labels (such as "medical service", "social security", "housing problem", etc.).

[0206] Result: N candidate labels are obtained, and subsequent steps will be further optimized based on these candidate labels. In the current business scenario, each candidate label is used to represent the matter category to which the input text belongs.

[0207] Step 2: confidence threshold determination.

[0208] Specific steps: the initial confidence of the Top-1 label output in step 1 is determined. If P(y1|x)≥τ, the Top-1 label is directly taken as the final target classification result.

[0209] Result: if the initial confidence of the Top-1 label is high enough, the text classification system can directly return the Top-1 label, thereby reducing subsequent processing and ensuring quick response to simple appeals; if the initial confidence of the Top-1 label is lower than the set threshold, the next step is entered, thereby ensuring accurate processing of complex appeals.

[0210] Step 3: combine the appeal description to construct the prompt word text related to the candidate label.

[0211] Specific steps: combine the appeal content, assign roles and tasks to the large language model, and construct the prompt word text: "You are a government service expert with 10 years of work experience. Please judge the following N candidate labels according to the citizen's appeal description, and output the most suitable candidate label. The candidate labels are: medical service, social security, housing problem."

[0212] Result: the prompt word text is obtained, and the input text and the prompt word text are combined for subsequent large language model analysis.

[0213] Step 4: Perform secondary focus analysis on the N candidate labels by a large language model (such as the general-purpose Q&A model), and evaluate the accuracy of each candidate label.

[0214] Specific steps: input the prompt word text and the original input text into the general-purpose Q&A model, so as to perform deep semantic analysis and classification optimization on the N candidate labels by the general-purpose Q&A model, and generate evaluation results for each candidate label (for example, the target confidence of “medical services” is 0.85, the target confidence of “housing problems” is 0.10, and the target confidence of “social security” is 0.05).

[0215] Result: the evaluation results for each candidate label are obtained, and the general-purpose Q&A model outputs the most suitable target candidate label (for example, “medical services”).

[0216] As can be seen, in the classification scenario of citizen appeal items, the RoBERTa-based text classification model can only provide N candidate labels (each candidate label corresponds to an item category), and the embodiment of the present application can select the most suitable target candidate label from the N candidate labels through secondary focus of the large language model, thereby improving the intelligence and efficiency of the government hotline platform. The embodiment of the present application can perform well when dealing with multi-item texts and complex semantic analysis through the introduction of the large language model and the optimization strategy of the prompt word text, has strong adaptability, and can be widely applied to other text processing fields.

[0217] Corresponding to the foregoing application function implementation method embodiment, the present application also provides a text classification device, an electronic device, a computer readable storage medium and corresponding embodiments.

[0218] Figure 5 is a structural schematic diagram of a text classification device according to an embodiment of the present application.

[0219] Referring to Figure 5 , the text classification device provided by the present application can include:

[0220] The preliminary classification module 510 is configured to obtain an input text, and call a text classification model to classify the input text by the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result includes N candidate labels, each candidate label is used to represent a category to which the input text belongs, and N is a positive integer;

[0221] The prompt word text construction module 520 is configured to construct a prompt word text by using the input text and the N candidate labels;

[0222] The secondary aggregation module 530 is configured to invoke the large language model to evaluate each candidate label according to the prompt word text by the large language model to obtain a target confidence of each candidate label.

[0223] The target candidate label selection module 540 is configured to select a target candidate label with the highest target confidence from the N candidate labels as the target classification result based on the target confidence of each candidate label.

[0224] In an embodiment, the initial classification result further includes an initial confidence of each candidate label; before the input text and the N candidate labels are used to construct the prompt word text, the device can further include:

[0225] The preferred candidate label selection module is configured to select a preferred candidate label with the highest initial confidence from the N candidate labels based on the initial confidence of each candidate label.

[0226] The confidence threshold judgment module is configured to determine whether the initial confidence of the preferred candidate label is greater than or equal to a set threshold.

[0227] The accurate processing module is configured to determine to perform the step of using the input text and the N candidate labels to construct the prompt word text if the initial confidence of the preferred candidate label is less than the set threshold.

[0228] In an embodiment, the device can further include:

[0229] The fast response module is configured to take the preferred candidate label as the target classification result if the initial confidence of the preferred candidate label is greater than or equal to the set threshold.

[0230] In an embodiment, the prompt word text construction module 520 can include:

[0231] The current business scenario determination submodule is configured to determine a current business scenario according to the input text.

[0232] The target template calling submodule is configured to call a target template matched with the current business scenario.

[0233] The data splicing submodule is configured to splice the input text and the N candidate labels into the target template to obtain the prompt word text.

[0234] In an embodiment, the secondary aggregation module 530 can include:

[0235] The data input submodule is configured to input the input text and the prompt word text into the large language model.

[0236] The deep semantic analysis submodule is configured to evaluate the matching degree of each candidate label in the prompt word text with the input text according to deep semantic analysis on the input text and the prompt word text by the large language model, so as to obtain the target confidence of each candidate label.

[0237] In an embodiment, the preliminary classification module 510 can include:

[0238] The word segmentation processing submodule is configured to perform word segmentation processing on the input text to obtain a plurality of subwords.

[0239] The preset mark insertion submodule is configured to input the input text after word segmentation processing into the text classification model, so as to insert a preset mark in a specific position of the input text by the text classification model.

[0240] The vector conversion submodule is configured to convert each subword into a subword embedding vector and convert the preset mark into a mark embedding vector by the text classification model.

[0241] The first encoding processing submodule is configured to perform encoding processing on each subword embedding vector by the text classification model to obtain an initial text representation of each subword embedding vector.

[0242] The aggregation representation submodule is configured to interact the mark embedding vector with each initial text representation by the text classification model to obtain a target mark embedding vector; the target mark embedding vector carries context information of the entire input text.

[0243] The second encoding processing submodule is configured to perform encoding processing on the target mark embedding vector by the text classification model to obtain a target text representation of the target mark embedding vector.

[0244] The preliminary classification submodule is configured to classify the input text based on the target text representation by the text classification model to obtain an initial classification result including N candidate labels and an initial confidence of each candidate label.

[0245] In an embodiment, before invoking the text classification model, the device can further include:

[0246] The sample acquisition module is configured to acquire a pre-trained text classification model, and acquire a test sample set and a labeled data set; the test sample set includes a plurality of test sample texts, and the labeled data set includes a real label labeled for each test sample text.

[0247] The model prediction module is configured to classify the plurality of test sample texts respectively by the text classification model to obtain a sample classification result of each test sample text; the sample classification result includes an optimal sample label of the test sample text.

[0248] The label matching module is configured to match the preferred sample labels belonging to the same test sample text with the true labels to obtain label matching results of the test sample texts;

[0249] The classification evaluation index calculation module is configured to calculate classification evaluation indexes of the text classification model according to the label matching results of the test sample texts, wherein the classification evaluation indexes include accuracy, precision and recall rate;

[0250] The model performance evaluation module is configured to determine whether the classification evaluation indexes are greater than or equal to preset indexes.

[0251] The model performance good evaluation module is configured to determine to execute the step of calling the text classification model if the classification evaluation indexes are greater than or equal to the preset indexes.

[0252] As can be seen from the example, the scheme provided in the present application obtains an input text and calls a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, the initial classification result includes N candidate labels, each candidate label is used to represent a category to which the input text belongs, and N is a positive integer; the input text and the N candidate labels are used to construct a prompt word text; a large language model is called to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label; and a target candidate label with the highest target confidence is selected from the N candidate labels as a target classification result based on the target confidence of each candidate label. The present application classifies the input text by introducing the BERT model, the BERT model can output more accurate N candidate labels, and then the N candidate labels are analyzed in depth by introducing the large language model, the large language model can perform more accurate result evaluation in a complex language scene, in addition, the prompt word text can effectively guide the large language model to focus on key semantic points, so that the large language model better understands the classification task, thereby effectively improving the accuracy and reliability of text classification.

[0253] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in details in the embodiments about the method, and thus will not be described in details here.

[0254] Figure 6 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0255] Referring to FIG. 6, Figure 6 The electronic device 600 includes a memory 610 and a processor 620.

[0256] The processor 620 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can be any conventional processor.

[0257] The memory 610 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device can be a read-and-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-and-write storage device or a volatile read-and-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 610 can include a combination of any computer-readable storage media, including various types of semiconductor memory chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 610 can include a read-and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include a carrier wave and a transient electronic signal transmitted through a wireless or wired transmission.

[0258] The memory 610 stores executable code, which, when processed by the processor 620, can cause the processor 620 to perform part or all of the above-mentioned methods.

[0259] Furthermore, the method according to the present application can also be implemented as a computer program or a computer program product, which comprises computer program code instructions for executing some or all of the steps of the above-mentioned method according to the present application.

[0260] Alternatively, the present application can also be implemented as a computer readable storage medium (or a non-transitory machine readable storage medium or a machine readable storage medium) having stored thereon executable codes (or computer programs or computer instruction codes) which, when executed by a processor of an electronic device (or a server, etc.), cause the processor to perform some or all of the steps of the above-mentioned method according to the present application.

[0261] The above has described the embodiments of the present application, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical application, or improvement to the technology in the market, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of text classification, characterized by, The method comprises: acquiring an input text and calling a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, and the initial classification result comprises N candidate labels, each candidate label being used to represent a category to which the input text belongs, and N being a positive integer; using the input text and the N candidate labels to construct a prompt word text; calling a large language model to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label; selecting a target candidate label with the highest target confidence from the N candidate labels as a target classification result based on the target confidence of each candidate label.

2. The method of claim 1, wherein, The initial classification result further comprises an initial confidence of each candidate label; Before the step of using the input text and the N candidate labels to construct a prompt word text, the method further comprises: selecting a preferred candidate label with the highest initial confidence from the N candidate labels based on the initial confidence of each candidate label; determining whether the initial confidence of the preferred candidate label is greater than or equal to a set threshold value; if the initial confidence of the preferred candidate label is less than the set threshold value, determining to perform the step of using the input text and the N candidate labels to construct a prompt word text.

3. The method of claim 2, wherein, The method further comprises: if the initial confidence of the preferred candidate label is greater than or equal to the set threshold value, taking the preferred candidate label as the target classification result.

4. The method of claim 1, wherein, The step of using the input text and the N candidate labels to construct a prompt word text comprises: determining a current business scenario according to the input text; calling a target template matched with the current business scenario; splicing the input text and the N candidate labels into the target template to obtain a prompt word text.

5. The method of claim 1, wherein, The step of evaluating each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label comprises: inputting the input text and the prompt word text into the large language model; evaluating the matching degree of each candidate label in the prompt word text with the input text according to the deep semantic analysis of the input text and the prompt word text by the large language model to obtain the target confidence of each candidate label.

6. The method of claim 2, wherein, The step of classifying the input text through the text classification model to obtain an initial classification result comprises: performing word segmentation processing on the input text to obtain a plurality of subwords; inputting the input text after word segmentation processing into the text classification model to insert a preset mark in a specific position of the input text through the text classification model; converting each subword into a subword embedding vector and converting the preset mark into a mark embedding vector through the text classification model; performing encoding processing on each subword embedding vector through the text classification model to obtain an initial text representation of each subword embedding vector; The target token embedding vector is obtained by interacting each of the token embedding vectors with each of the initial text representations through the text classification model; the target token embedding vector carries context information of the entire input text; The target text representation of the target token embedding vector is obtained by encoding the target token embedding vector through the text classification model; The input text is classified based on the target text representation through the text classification model to obtain an initial classification result including N candidate labels and initial confidence of each candidate label.

7. The method of claim 1, wherein, Before the step of invoking the text classification model, the method further includes: obtaining a pre-trained text classification model, a test sample set and a labeled data set; the test sample set includes a plurality of test sample texts, and the labeled data set includes a true label labeled for each test sample text; The sample classification result of each test sample text is obtained by classifying a plurality of test sample texts through the text classification model; the sample classification result includes an optimal sample label of the test sample text; The optimal sample label and the true label belonging to the same test sample text are matched to obtain a label matching result of each test sample text; The classification evaluation index of the text classification model is calculated according to the label matching result of each test sample text; the classification evaluation index includes accuracy, precision and recall; It is judged whether the classification evaluation index is greater than or equal to a preset index; If the classification evaluation index is greater than or equal to the preset index, it is determined to execute the step of invoking the text classification model.

8. A text classification apparatus characterized by comprising: The device includes: An initial classification module is configured to obtain an input text and invoke a text classification model to classify the input text through the text classification model to obtain an initial classification result; wherein the text classification model is a BERT model, the initial classification result includes N candidate labels, each candidate label is used to represent a category to which the input text belongs, and N is a positive integer; A prompt word text construction module is configured to construct a prompt word text using the input text and N candidate labels; A secondary aggregation module is configured to invoke a large language model to evaluate each candidate label according to the prompt word text through the large language model to obtain a target confidence of each candidate label; A target candidate label selection module is configured to select a target candidate label with the highest target confidence from N candidate labels as a target classification result based on the target confidence of each candidate label.

9. An electronic device, comprising: It includes: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to execute the method of any one of claims 1-7.

10. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method of any one of claims 1-7.

Images