Class label determination method and device, computer equipment and labeling model training method

Abstract

The invention relates to a category label determination method and device, computer equipment, a storage medium and a computer program product, and a labeling model training method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: determining a text data set; predicting a first probability value corresponding to the first text according to the first text; according to a difference value between the first probability values and a difference value between the first probability values and a first reference probability value, first classification evaluation index data of the first text is obtained through calculation; screening out a first text of which the first classification evaluation index data is smaller than a preset threshold value, and updating a labeled category label of the first text; and if the first text with the labeled category label inconsistent with the updated category label exists in the text data set, replacing the labeled category label of the first text with the updated category label. By adopting the method, the efficiency and accuracy of determining the category labels can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, computer equipment, storage medium, computer program product for determining a category label, and a method, device, computer equipment, storage medium, computer program product for training a labeling model. Background Art

[0002] With the development of science and technology, machine learning and artificial intelligence technologies have been widely used. Labeled data refers to data used to train and verify models, and is the basis of artificial intelligence algorithms. Artificial intelligence algorithms require a large amount of labeled data for training, and the quality and quantity of labeled data directly affect the performance and accuracy of artificial intelligence algorithms.

[0003] In traditional technologies, data needs to be manually labeled, classified, annotated or marked to obtain labeled data. Since the amount of labeled data in different fields is limited and model training requires a large amount of labeled data, a large amount of manpower is usually required for manual labeling for different application scenarios. The labeled data obtained by this method has low labeling efficiency, and the accuracy and stability of the labeling are difficult to control. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for determining category labels and a training method, device, computer equipment, storage medium and computer program product for labeling models that can improve efficiency and accuracy in response to the above-mentioned technical problems.

[0005] In a first aspect, the present disclosure provides a method for determining a category label, comprising:

[0006] Determine a text data set, wherein the text data set includes a first text with annotated category labels;

[0007] Predicting, according to the first text, a first probability value corresponding to the first text;

[0008] Calculate the first classification evaluation index data of the first text according to the difference between the first probability values ​​and the difference between the first probability value and the first reference probability value;

[0009] Filter out the first text whose first classification evaluation index data is less than a preset threshold and update the marked category label of the first text;

[0010] If there is a first text in the text data set whose annotated category label is inconsistent with the updated category label, the annotated category label of the first text is replaced with the updated category label.

[0011] In a second aspect, the present disclosure further provides a method for training a labeling model, comprising:

[0012] Inputting a first training text data set including first training texts with annotated category labels into an initial annotation model for category prediction, and outputting a second probability value corresponding to the first training text;

[0013] Determining a classification loss function according to the marked category label and the second probability value;

[0014] Determine, according to the second probability value, a first similarity between every two first training texts in the training text data set and a similarity label between every two first training texts in the training text data set;

[0015] Determining a contrast loss function according to the first similarity and the similarity label;

[0016] The initial annotation model is trained according to the classification loss function and the contrast loss function.

[0017] In a third aspect, the present disclosure further provides a device for determining a category label, comprising:

[0018] A determination module, configured to determine a text data set, wherein the text data set includes a first text with a marked category label;

[0019] A prediction module, configured to predict a first probability value corresponding to the first text according to the first text;

[0020] A calculation module, used for calculating the first classification evaluation index data of the first text according to the difference between the first probability values ​​and the difference between the first probability value and the first reference probability value;

[0021] A screening module, used to screen out the first text whose first classification evaluation index data is less than a preset threshold and update the marked category label of the first text;

[0022] The replacement module is used to replace the marked category label of the first text with the updated category label if there is a first text in the text data set whose marked category label is inconsistent with the updated category label.

[0023] In one embodiment, the computing module comprises:

[0024] A first determining submodule, configured to determine, according to a maximum probability value and a second maximum probability value in the first probability values, a first difference between the maximum probability value and the second maximum probability value, and a second difference between the maximum probability value and a first reference probability value;

[0025] A summing module is used to perform a sum operation on the first difference and the second difference to obtain first classification evaluation index data of the first text.

[0026] In one embodiment, the prediction module includes:

[0027] The prediction submodule is used to input the first text into the annotation model for category prediction and output a first probability value corresponding to the first text.

[0028] In one embodiment, the determining module includes:

[0029] An acquisition submodule, used to acquire a second text that has been annotated with an initial category label;

[0030] A second determination submodule, configured to determine a marked category label of the first text according to an association relationship between the second text and the first text;

[0031] The first obtaining module is used to obtain the text data set according to the first text and the marked category label of the first text.

[0032] In one embodiment, the second determining submodule includes:

[0033] An acquisition unit, configured to acquire keywords of the second text;

[0034] A determination unit is used to determine the initial category label of the second text as the marked category label of the first text if the keyword matches the first text and the text similarity between the second text and the first text is greater than a preset threshold.

[0035] In a fourth aspect, the present disclosure further provides a training device for a labeling model, comprising:

[0036] An input module, used for inputting a first training text data set consisting of first training texts with annotated category labels into an initial annotation model for category prediction, and outputting a second probability value corresponding to the first training text;

[0037] A first determination module, configured to determine a classification loss function according to the marked category label and the second probability value;

[0038] A second determination module is used to determine a first similarity between every two first training texts in the training text data set and a similarity label between every two first training texts in the training text data set according to the second probability value;

[0039] A third determination module, configured to determine a contrast loss function according to the first similarity and the similarity label;

[0040] A training module is used to train the initial annotation model according to the classification loss function and the contrast loss function.

[0041] In one embodiment, the third determination module includes:

[0042] A second obtaining module is used to obtain a first component of a contrast loss function according to the first similarity, the similarity label and the number of first training text pairs consisting of every two first training texts;

[0043] A third obtaining module, configured to obtain a second component of a contrast loss function according to the first similarity, the similarity label and a preset threshold of the first similarity;

[0044] The third determination submodule is used to determine a contrast loss function according to the first component and the second component.

[0045] In one embodiment, the device further comprises:

[0046] An output submodule, used for inputting a second training text data set consisting of second training texts with annotated category labels into the trained initial annotation model for category prediction, and outputting a third probability value that the second training text belongs to the annotated category;

[0047] a calculation submodule, configured to calculate the second classification evaluation index data of the second training text according to the third probability value and the difference between the third probability value and the second reference probability value;

[0048] A screening submodule, used to screen out the second training texts whose second classification evaluation index data is less than a preset threshold and update the marked category labels of the second training texts;

[0049] A replacement submodule, configured to replace the marked category label of the first training text with the updated category label if the first training text data set contains a first training text whose marked category label is inconsistent with the updated category label of the second training text;

[0050] The training submodule is used to train the trained initial annotation model using the updated first training text.

[0051] In a fifth aspect, an embodiment of the present disclosure further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of any one of the methods in the embodiments of the present disclosure when executing the computer program.

[0052] In a sixth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0053] In a seventh aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0054] The above-mentioned method, apparatus, computer equipment, storage medium and computer program product for determining category labels, when determining the category label, determine a text data set including a first text with annotated category labels, and predict a first probability value that the first text belongs to the annotated category based on the first text, and calculate the first classification evaluation index data of the first text based on the difference between the first probability values ​​and the difference between the first probability value and the first benchmark probability value, so as to quickly and accurately obtain the evaluation index data that can be used to evaluate the difficulty of text classification, thereby ensuring the accuracy of subsequent screening; based on the first classification evaluation index data, screen and update the annotated category labels of the first text, replace the annotated category labels that are inconsistent before and after the update in the text data set, realize the update of the category labels of the "difficult to classify" texts, ensure the accuracy of the obtained category labels, and at the same time, take into account the efficiency of text data category labeling and reduce the workload of manual labeling; by screening and updating the text with the first classification evaluation index data, the "difficult to classify" samples can be adjusted in time, taking into account the efficiency and accuracy of automatic labeling, and being suitable for more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is a flow chart of a method for determining a category label in one embodiment;

[0057] Figure 2 is a flow chart of a method for determining a category label in another embodiment;

[0058] Figure 3 A schematic diagram of a flow chart of a method for determining a text data set in one embodiment;

[0059] Figure 4 A schematic diagram of a flow chart of a method for training a labeling model in one embodiment;

[0060] Figure 5 is a flow chart of a method for determining a category label in another embodiment;

[0061] Figure 6 A schematic diagram of a flow chart of a training method for a labeling model in another embodiment;

[0062] Figure 7 is a structural block diagram of a device for determining a category label in one embodiment;

[0063] Figure 8 is a structural block diagram of a training device for a labeling model in an embodiment;

[0064] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0066] Data labeling usually refers to adding classification labels to voice, pictures, text, video and other data required for model training or verification, so as to convert the data into machine-recognizable information. In the field of artificial intelligence algorithms, a large amount of labeled data is usually required for model training. The quality and quantity of labeled data directly affect the performance and accuracy of artificial intelligence algorithms. When it is necessary to train artificial intelligence models such as text classification models, a large amount of text labeled with category labels is required as training data. In related technologies, in order to ensure the quality of training data, it is necessary to manually label the text with category labels. However, the manual method consumes high manpower and time costs, and the labeling standards and labeling results will be unstable due to differences in understanding and fatigue of the labelers. The efficiency is low and the labeling accuracy cannot be guaranteed.

[0067] In order to solve the above technical problems, when determining the category label, the embodiment of the present disclosure first obtains the first text with the labeled category label and predicts the first probability value corresponding to the first text; determines the first classification evaluation index data of the first text based on the first probability value, and screens the first text and updates the labeled category label according to the first classification evaluation index data, so that the labeled category label of the first text can be optimized based on the first classification evaluation index data of the first text. Without manual labeling of the category label, the labeled text can be obtained efficiently and accurately, thereby improving the efficiency and accuracy of text category labeling.

[0068] The following is an introduction to some nouns and technical terms involved in the present disclosure. It should be noted that the purpose of explaining the terms described in the present disclosure is to facilitate a more accurate understanding of the innovative technical content of the present solution.

[0069] A text data set generally refers to a set consisting of text data. When it is necessary to determine a category label for a text, the set consisting of the text to be processed that needs to be determined for a category label can be called a text data set.

[0070] The first text may include text for which a category label needs to be determined in some embodiments of the present disclosure, wherein the first text is text that has been marked with a category label.

[0071] The labeled category label generally refers to the label corresponding to the first text before the category label is determined. The labeled category label may include labels obtained by similarity matching algorithms, retrieval algorithms, labeling models, etc.

[0072] The first probability value may be a probability value obtained by predicting the first text that the first text belongs to different text categories. The first probability value may include one or more probability values.

[0073] The first benchmark probability value may be a probability value pre-set according to an actual application scenario, and the first classification evaluation index data for measuring the difficulty of text classification may be obtained in combination with the first probability value.

[0074] The first classification evaluation index data can be determined based on the probability values ​​of the text belonging to different categories, and is used to measure the difficulty of text classification.

[0075] A prediction probability vector is usually a vector used in statistics and machine learning to represent the probability of an event or category occurring. In classification problems, when a model is trained and new samples are predicted, a probability vector is usually obtained, in which each element represents the probability of occurrence of the corresponding category.

[0076] The BERT (Bidirectional Encoder Representations from Transformers) model is usually a pre-trained language model based on the Transformer architecture, which learns language representation from a large amount of text data through large-scale unsupervised training. The BERT model is a bidirectional, deep neural network model that can capture the complex relationship between words in a sentence and provides powerful language representation capabilities for various text-related tasks.

[0077] K-fold cross validation is a commonly used model evaluation method to evaluate the performance and generalization ability of machine learning models. In K-fold cross validation, the dataset is divided into K subsets of similar size, called folds. Then, the model is trained and validated K times, with K-1 folds used for training each time and the remaining fold used to validate the model. Finally, the average of the K validation results is used as the performance evaluation indicator of the model. K-fold cross validation is a commonly used model evaluation method that can provide more accurate and reliable model performance evaluation and help select the best machine learning model.

[0078] Boolean retrieval is usually an algorithm in the field of information retrieval that extracts text containing key information (keywords, keywords, etc.). It can be used to search for data containing the same key information in the data to be annotated for keywords extracted from the produced samples.

[0079] The BM25 algorithm is usually an algorithm used in the field of information retrieval to calculate the similarity score between sentences and documents. Its general formula is: ,in, Represents a statement, Represents a word in a sentence. Represents a search document, that is, a question template in the matching template. Represents the word weight, and its formula is , where N represents the number of problem templates in the matching template, To include Therefore, for a word in a sentence, the more question templates contain the word, the smaller the IDF is, that is, the less important the word is.

[0080] In one embodiment, Figure 1 As shown, a method for determining a category label is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0081] Step S110, determining a text data set, wherein the text data set includes a first text with annotated category labels;

[0082] Exemplarily, a text data set may be a set of texts that need to be assigned a category label. A first text may be a text that needs to be assigned a category label. A text data set is determined, wherein the text data set includes a first text that needs to be assigned a category label, wherein the first text is a text that has been labeled with a category label. The category of the first text may be determined based on an actual business scenario, and different business scenarios may correspond to different categories of text data. Business scenarios may include, but are not limited to, sentiment analysis, news classification, spam filtering, text topic identification, etc., and may be specifically set based on actual application scenarios. In one example, the labeled category label of the first text is obtained by rough labeling, that is, the accuracy of the labeled category label is low.

[0083] In one example, the annotated category label of the first text may be obtained after automatic annotation by a preset method. In some possible embodiments, the annotated category label of the first text may be obtained by text retrieval and similarity comparison; the annotated category label of the first text may be obtained by an annotation model, which may be determined according to the actual application scenario.

[0084] Step S120, predicting a first probability value corresponding to the first text according to the first text;

[0085] In the disclosed embodiment, according to the first text, the first probability value corresponding to the first text is predicted. Exemplarily, the first text may belong to different categories, and according to the first text, the first probability value of the first text belonging to different categories can be determined. In one example, the text information of the first text can be analyzed to obtain the first probability value of the first text belonging to different categories, and the first probability value may include one or more probability values. In a possible application scenario, for example, a customer service robot determines the category of the question text sent by the user, and the first text can be the text corresponding to the question sent by the user. The category may include but is not limited to ticket booking, information inquiry, problem feedback, etc. After obtaining the first text, the text information of the first text can be analyzed to obtain the probability value A of the first text belonging to the ticket booking category, the probability value B of the first text belonging to the information inquiry category, and the probability value C of the first text belonging to the problem feedback category. The probability values ​​A, B, and C are the first probability values ​​of the first text belonging to different categories.

[0086] Optionally, the first text can be processed by a preset text information analysis method to predict first probability values ​​that the first text belongs to different categories; the first text can also be predicted by a preset annotation model to obtain first probability values ​​that the first text belongs to different categories, which can be specifically determined according to actual application scenarios.

[0087] Step S130, calculating first classification evaluation index data of the first text according to the difference between the first probability values ​​and the difference between the first probability value and a first reference probability value;

[0088] In the disclosed embodiment, first classification evaluation index data of the first text is calculated based on the difference between the first probability values ​​and the difference between the first probability value and the first benchmark probability value. The first classification evaluation index data is used to measure the difficulty of text classification. Exemplarily, the first probability value includes multiple probability values, which respectively correspond to the first probability values ​​of the first text belonging to different categories; the first benchmark probability value can be determined in advance according to the actual application scenario. The larger the difference between the first probability value and the first benchmark probability value, the lower the difficulty of classifying the first text into the category corresponding to the first probability value. Therefore, the first classification evaluation index data obtained by the difference between the first probability value and the first benchmark probability value can be accurately used to measure the difficulty of text classification, filter out texts with higher classification difficulty, update the labels of the filtered texts, and improve the accuracy of the text annotation labels.

[0089] In one example, the first probability value includes multiple probability values, and the difference between the first probability values ​​can be selecting two probability values ​​from the multiple first probability values ​​and calculating the difference between the two; the difference between the first probability values ​​can also be the difference between the multiple first probability values. The method of selecting the probability value can be determined according to the actual application scenario, for example, the largest first probability value and the second largest first probability value are selected according to the size of the probability value to obtain the difference.

[0090] When calculating the first classification evaluation index data according to the determined difference, the calculation method can be determined according to the actual application scenario. In one example, the sum of the determined difference can be directly used as the first classification evaluation index data; in another example, the weights corresponding to different types of difference can also be determined, and the determined weights are weighted to obtain the first classification evaluation index data. In one example, the first classification evaluation index data can be in the form of a score, and the higher the score, the lower the classification difficulty of the first text corresponding to the first classification evaluation index data.

[0091] Step S140, screening out the first text whose first classification evaluation index data is less than a preset threshold and updating the marked category label of the first text;

[0092] In the disclosed embodiment, after obtaining the first classification evaluation index data, the first text is screened. Exemplarily, the smaller the first classification evaluation index data is, the greater the classification difficulty of the corresponding first text can be considered. Therefore, the first text whose first classification evaluation index data is less than the preset threshold is screened out. The preset threshold can be an index value obtained according to the actual application scenario. When the first classification evaluation index data is less than the preset threshold, it can be considered that the classification difficulty of the corresponding first text is relatively large, and the probability of the annotated category label being wrong is relatively large, and it needs to be updated. In an example, when the first classification evaluation index data is in the form of a score, the preset threshold can also be a score set according to the actual application scenario.

[0093] In one example, the method for updating the annotated category label can be set according to the actual application scenario. For example, the first text selected can be re-annotated by the annotation model to obtain an updated category label; the first text selected can be re-annotated manually to obtain an updated category label; or other methods capable of labeling can be used to annotate to obtain an updated category label.

[0094] Step S150: If there is a first text in the text data set whose annotated category label is inconsistent with the updated category label, the annotated category label of the first text is replaced with the updated category label.

[0095] Exemplarily, if there is a first text in the text data set whose labeled category label is inconsistent with the updated category label, it means that the first classification evaluation index data meets the screening conditions and the first text has undergone label update. At this time, the reliability of the updated category label is greater than the labeled category label before the update. Therefore, the labeled category label of the first text is replaced with the updated category label.

[0096] In the disclosed embodiment, when determining a category label, a text data set including a first text with an annotated category label is determined, and a first probability value that the first text belongs to the annotated category is predicted based on the first text, and first classification evaluation index data of the first text is calculated based on the difference between the first probability values ​​and the difference between the first probability value and the first benchmark probability value, so that evaluation index data that can be used to evaluate the difficulty of text classification can be obtained quickly and accurately, thereby ensuring the accuracy of subsequent screening; based on the first classification evaluation index data, the annotated category labels of the first text are screened and updated, and the annotated category labels that are inconsistent before and after the update in the text data set are replaced, thereby realizing the update of the category labels of the "difficult to classify" texts, ensuring the accuracy of the obtained category labels, and at the same time, taking into account the efficiency of text data category annotation and reducing the workload of manual annotation; by screening and updating the text with the first classification evaluation index data, the "difficult to classify" samples can be adjusted in time, taking into account the efficiency and accuracy of automatic annotation, and being suitable for more application scenarios.

[0097] In one embodiment, Figure 2 As shown, the first classification evaluation index data of the first text is calculated based on the difference between the first probability values ​​and the difference between the first probability value and the first reference probability value, including:

[0098] Step S131, determining a first difference between the maximum probability value and the second maximum probability value, and a second difference between the maximum probability value and a first reference probability value according to a maximum probability value and a second maximum probability value in the first probability values;

[0099] Step S132, performing a sum operation on the first difference and the second difference to obtain first classification evaluation index data of the first text.

[0100] Exemplarily, the maximum probability value may be the probability value with the largest value among the first probability values, and the second-largest probability value may be the probability value with the second-largest value among the first probability values. The maximum probability value and the second-largest probability value among the first probability values ​​are determined to represent the maximum possible category probability value and the second-largest possible category probability value in the predicted category to which the first text belongs, respectively. The first difference is the difference between the maximum probability value and the second-largest probability value, wherein the larger the first difference is, the greater the difference between the prediction that the first text belongs to the maximum possible category and the second-largest possible category is, that is, the lower the text classification difficulty is. Therefore, the first classification evaluation index data obtained through the first difference can accurately measure the classification difficulty of the text, filter out texts with higher classification difficulty, update the filtered texts, and improve the accuracy of the text annotation labels.

[0101] The second difference is the difference between the maximum probability value and the first reference probability value, wherein the larger the second difference is, the clearer the first text classification result is and the lower the classification difficulty is. The first reference probability value is set according to the actual application scenario, for example, it can be set to 0.5.

[0102] The first difference and the second difference are summed to obtain first classification evaluation index data of the text. In an example, the first classification evaluation index data may be set in the form of a score. The lower the score, the higher the difficulty of text classification.

[0103] In one example, the predicted first probability value is the prediction result, which can be represented by a prediction probability vector, and the first classification evaluation index data can be set in the form of a score, that is, an outlier score. , where i represents the first predicted text and N represents the predicted text category. The outlier score is calculated based on the first probability value in the prediction result, as shown in formula (1).

[0104] (1)

[0105] Among them, score is the outlier score, that is, the first classification evaluation index data, represents a set consisting of each value of the probability vector, that is, the first probability value corresponding to the first text, represents the maximum probability value with the largest median value among the first probability values, It represents the second largest probability value in the first probability value. In this formula, the first benchmark probability value is set to 0.5. The higher the probability value, the higher the confidence that the text belongs to the category. For the predicted probability vector of a text, the class n represented by the maximum probability value is the class to which it belongs. The larger the probability value, the higher the confidence. In an example, 0.5 is used as the reference value, which is the first benchmark probability value. At this time, the distance between the maximum probability value and the reference value is , that is, the second difference is obtained; if the difference between the maximum probability value and the second maximum probability value is larger, the credibility of the text belonging to the category represented by the maximum probability value is higher, and the distance between the two probabilities is measured as , that is, the first difference is obtained. The first difference and the second difference are added to obtain the outlier score, that is, the larger the outlier score, the lower the difficulty of text classification and the better the classification effect. In one example, the value range of the second difference is (-0.5, 0.5], and the value range of the first difference is [0, 1]. After addition, the value range of the outlier score is (-0.5, 0.5]. At this time, the preset threshold can be set to any value in the value range of the outlier score. The smaller the preset threshold, the fewer first texts are screened out. For example, the preset threshold can be set to 0, and texts with outlier scores less than 0 are screened out and the labels are updated. In some possible implementations, the preset threshold can be flexibly adjusted according to the actual application scenario to meet different needs. For example, when the preset threshold is set to a larger value, such as 0.2, more first texts are screened out, and the accuracy of the updated category label is higher; when the preset threshold is set to a smaller value, such as -0.2, less first texts are screened out, and the efficiency of text label update determination is higher.

[0106] In the disclosed embodiment, a first difference representing confidence can be determined based on the maximum probability value and the second maximum probability value, and a second difference representing an edge sampling score can be determined based on the difference between the maximum probability value and the first baseline probability value. The first classification evaluation index data of the text can be quickly and accurately calculated by combining the first difference and the second difference, thereby achieving a quantitative representation of the difficulty of text classification. By combining confidence and edge sampling, the accuracy of the first classification evaluation index data is guaranteed, and the accuracy of text screening based on the first classification evaluation index data is improved, thereby effectively improving the accuracy of the category label of the updated first text.

[0107] In one embodiment, predicting a first probability value corresponding to the first text according to the first text includes:

[0108] The first text is input into a labeling model for category prediction, and a first probability value corresponding to the first text is output.

[0109] In the disclosed embodiment, when determining the first probability value of the first text, category prediction can be performed through the annotation model. The annotation model is a model for predicting the category of the text. The annotation model can analyze and process the input text to obtain the probability value that the text belongs to different categories, that is, the first probability value. Exemplarily, the first text is input into the annotation model capable of performing category prediction to perform category prediction, and the first probability value that the first text belongs to the annotated category is output. In one example, the annotation model can be constructed and trained based on a preset natural language processing model. The natural language processing model may include but is not limited to a BERT (Bidirectional Encoder Representations from Transformers) model, etc. For example, text conversion can be performed based on the Encoder part of BERT, wherein the Encoder part of the BERT model refers to the encoder part in its Transformer structure, which is used to convert the input text into a vector representing the semantics of the text.

[0110] In one example, the annotation model can be obtained by training based on a text data set. For example, the initial model is iteratively trained based on the text data set until the output accuracy of the initial model meets the requirements, thereby obtaining the annotation model, wherein the iterative training can be performed by K-fold cross-training. When the first probability value is obtained, the first probability value can be determined by the predicted probability value corresponding to the validation set in the K-fold cross-training process.

[0111] In one example, a text data set is grouped and model iteration training is performed. Exemplarily, for each group of first texts, other groups of texts except the group of texts are used as training samples for model training, and the target initial annotation model corresponding to the group of texts is obtained based on the corresponding relationship between other groups of texts and the annotated category labels. The group of texts is input into the target initial classification model to obtain a prediction result, which includes a first probability value corresponding to the group of texts. After executing the above process for each group of texts, the first probability value corresponding to the first text in the text data set is obtained. In one example, when the text data set is grouped, the text data can be divided into K groups of data according to the actual application scenario, and K-1 parts are used for training each time, and the remaining part is tested to obtain a prediction result. After K training and prediction, each first text in the text data set is predicted as test data to obtain a corresponding prediction result.

[0112] The disclosed embodiment performs category prediction based on the annotation model and outputs a first probability value, which can quickly and accurately obtain the first probability value, thereby ensuring the accuracy and efficiency of subsequent calculation of the first classification evaluation index data and screening and updating labels, and is suitable for more application scenarios; it further reduces manual workload and improves the efficiency of category label determination.

[0113] In one embodiment, determining the text data set includes:

[0114] Get the second text with the initial category label;

[0115] Determining a marked category label of the first text according to the association relationship between the second text and the first text;

[0116] The text data set is obtained according to the first text and the marked category label of the first text.

[0117] In the disclosed embodiment, a text data set can be determined by a second text that has been annotated with an initial category label. The second text is a text that has been annotated with an initial category label in advance, wherein the initial category label can be obtained by manual annotation or by annotation with a preset classification model, and can be determined specifically according to the actual application scenario. In one example, the annotated initial category label of the second text is a category label that has been determined, that is, the accuracy of the annotated initial category label meets the conditions.

[0118] According to the association relationship between the second text and the first text, the annotated category label of the first text is determined. Exemplarily, the association relationship between the second text and the first text can be determined by keyword search, similar text matching, etc. According to the association relationship between the second text and the text content of the first text, the initial category label of the second text with a higher degree of association with the first text can be determined as the annotated category label of the first text.

[0119] In the disclosed embodiment, a first text can be pre-annotated through a second text that has been annotated with an initial category label, thereby obtaining a text data set. Based on the association between the second text and the first text, the first text can be quickly roughly classified to obtain the annotated category label of the first text, thereby reducing the difficulty of determining the subsequent category label, reducing the manual workload, ensuring the efficiency of determining the subsequent category label, and being suitable for more application scenarios.

[0120] In one embodiment, determining the annotated category label of the first text according to the association relationship between the second text and the first text includes:

[0121] Obtaining keywords of the second text;

[0122] If the keyword matches the first text, and the text similarity between the second text and the first text is greater than a preset threshold, the initial category label of the second text is determined as the marked category label of the first text.

[0123] In an embodiment of the present disclosure, the annotated category label of the first text can be determined based on the keyword and text similarity. Exemplarily, the keyword of the second text is obtained. In one example, the keyword can be extracted by a keyword retrieval algorithm, and the keyword retrieval algorithm includes but is not limited to Boolean retrieval (BOOL retrieval) and the like. If the keyword matches the first text, and the text similarity between the second text and the first text is greater than a preset threshold, the initial category label of the second text is determined as the annotated category label of the first text, wherein the preset threshold can be a similarity threshold determined according to an actual application scenario. When it is greater than the preset threshold, it can be considered that the similarity between the second text and the first text is high, and the initial category label of the second text can be determined as the annotated category label of the first text.

[0124] Exemplarily, the text similarity between the first text and the second text may be determined by a preset similarity calculation algorithm. The preset similarity calculation algorithm may include but is not limited to the BM25 algorithm, etc., and may be determined specifically according to an actual application scenario.

[0125] In one example, if Figure 3 As shown, first make a second text. For example, for each category label, make a small amount of second text annotated with the initial category label, and use keywords, regular rules, etc. to sort out the characteristics and key information of each category. For example, when the customer service robot judges the user's question, the model needs to judge the type of question according to the user's question, and then it can recommend the correct answer to the user; in the ticket booking category, if the user inputs: I want to book a train ticket; how much is the direct flight from Beijing to Shanghai, it should belong to the ticket booking category, and when the user inputs: what delicious noodle restaurants are nearby, what is the weather like today, it does not belong to the ticket booking category. In one example, a small amount of text can be expanded based on experience or data augmentation tools to obtain a second text annotated with the initial category label.

[0126] The second text is used for keyword extraction to perform BOOL retrieval on the unlabeled data (i.e., the first text). At the same time, the second text is used to retrieve the unlabeled data in parallel through the BM25 algorithm, and the adoption ratio or quantity is defined. For example, the number of BM25 recalls for each example sample in each category = the number of unlabeled data / (the number of category labels+1) / the number of category example samples, where the number of category labels plus 1 takes into account other categories. The unlabeled data retrieved through BOOL retrieval and BM25 are merged and pre-labeled with the initial category label, that is, the first text is labeled with the initial category label to obtain the first text labeled with the category label, forming a text data set.

[0127] The disclosed embodiment combines the keywords of the second text and the similarity between the second text and the first text to determine the annotated category labels of the first text, and can quickly and accurately obtain roughly annotated category labels. The annotated category labels of the first text can be automatically determined based on a small amount of second text, which effectively reduces the workload of manual annotation, improves annotation efficiency, and is suitable for more application scenarios.

[0128] In another embodiment, Figure 4 As shown, a training method for a labeling model is also provided, including:

[0129] Step S410, inputting a first training text data set including first training texts with annotated category labels into an initial annotation model for category prediction, and outputting a second probability value corresponding to the first training text;

[0130] Step S420, determining a classification loss function according to the marked category label and the second probability value;

[0131] Step S430, determining a first similarity between every two first training texts in the training text data set and a similarity label between every two first training texts in the training text data set according to the second probability value;

[0132] Step S440, determining a contrast loss function according to the first similarity and the similarity label;

[0133] Step S450: training the initial annotation model according to the classification loss function and the contrast loss function.

[0134] The first training text is a text with a marked category label, which is used to train the annotation model. The second probability value is a probability value that the first training text output by the model belongs to a different text category. Exemplarily, a first training text data set consisting of first training texts with marked category labels is input into the initial annotation model for category prediction to obtain a second probability value corresponding to the first training text, wherein the second probability value may include probability values ​​that the first training text belongs to different categories, and the second probability value may include one or more probability values.

[0135] In one example, a classification loss function can be determined based on the difference between the labeled category label and the category corresponding to the maximum probability value in the second probability value. The classification loss function may include but is not limited to a cross entropy loss function, a mean square error loss function, a perceptual error loss function, etc., which can be determined based on the actual application scenario. Among them, the category corresponding to the maximum probability value in the second probability value can be considered as the category to which the text output by the model belongs.

[0136] Optionally, the training text data sets are grouped in pairs to determine the first similarity between every two first training texts in the training text data set and the similarity labels between every two first training texts in the training text data set, wherein the similarity labels are labels of whether the predicted categories of the two first training texts are similar, and the predicted category can be the category corresponding to the maximum probability value among the second probability values ​​of the first training texts.

[0137] Based on the first similarity and the similar labels, a contrast loss function can be determined. In one example, the contrast loss function can be calculated in combination with a preset margin value. The initial annotation model is trained according to the classification loss function and the contrast loss function. Exemplarily, the target loss function can be determined based on the sum of the classification loss function and the contrast loss function, and the training parameters of the model can be adjusted so that the target loss function value is less than a preset value. Among them, the preset value can be set according to the actual application scenario. When the target loss function value is less than the preset value, it can be considered that the model output result is more accurate at this time.

[0138] The disclosed embodiment combines the classification loss function and the contrast loss function to optimize the model training, which can better predict the text as the correct category while amplifying the errors of difficult-to-classify samples, thereby ensuring the effect of model training. It can then be used to more quickly and accurately screen out difficult-to-classify texts when determining category labels, and is suitable for more application scenarios.

[0139] In one embodiment, determining the contrast loss function according to the first similarity and the similarity label includes:

[0140] Obtaining a first component of a contrast loss function according to the first similarity, the similarity label, and the number of first training text pairs consisting of every two first training texts;

[0141] Obtaining a second component of the contrast loss function according to the first similarity, the similarity label, and a preset threshold of the first similarity;

[0142] A contrast loss function is determined according to the first component and the second component.

[0143] In the embodiment of the present disclosure, when determining the contrast loss function, a first component of the contrast loss function is obtained according to the first similarity, similar labels and the number of text pairs. A second component of the contrast loss function is obtained according to the first similarity, similar labels and a preset threshold of the first similarity.

[0144] In one example, if the similar labels are different, the proportions of the first component and the second component in the corresponding contrast loss function may be different. For example, the similar labels may be the same or different. If the similar labels are the same, the weight of the first component is increased, and if the similar labels are different, the weight of the second component is increased. According to the first component and the second component, the contrast loss function is determined.

[0145] In one example, the classification loss function and the contrast loss function are shown in formula (2).

[0146] (2)

[0147] In this formula, the classification loss function is set to the cross entropy loss function, represents the cross entropy loss function, n represents the number of categories, that is, the number of text categories, Represents the true label of the i-th category, that is, the labeled category label; It indicates the probability that the model predicts that this text belongs to the i-th category, that is, the second probability value; represents the contrast loss function, M represents the number of all possible text pair combinations in the first training text, that is, the number of the first training text pairs, is the label of whether the mth pair of texts is similar (y is equal to 1 if it is predicted to be of the same type, otherwise it is 0), that is, the similarity label of the text pair, is the distance between the mth pair of texts, that is, the first similarity between the text pairs, and margin is the marginal value, which is a pre-set threshold, that is, the preset threshold of the first similarity, usually indicating the boundary of similarity. Cross entropy loss is mainly used to measure the difference between the predicted value and the true value of the text, and contrast loss is used to measure the distance between texts predicted to be of the same category and the distance between texts of different predicted categories. By combining the two losses as the model optimization target, the model can better predict the text as the correct category while amplifying the errors of difficult-to-classify samples, which is conducive to screening them out through the calculation of the second classification evaluation index data when calculating the prediction probability of unknown text.

[0148] The disclosed embodiment determines a contrast loss function based on the first component and the second component, which can ensure the accuracy of the model trained by the contrast loss function while amplifying the errors of difficult-to-classify samples, thereby ensuring that difficult-to-classify texts are screened out more quickly and accurately when determining category labels, and is suitable for more application scenarios.

[0149] In one embodiment, the method further comprises:

[0150] Inputting a second training text data set consisting of second training texts with annotated category labels into the trained initial annotation model for category prediction, and outputting a third probability value that the second training text belongs to the annotated category;

[0151] Obtain second classification evaluation index data of the second training text according to the third probability value and the difference between the third probability value and the second reference probability value;

[0152] Screen out the second training text whose second classification evaluation index data is less than a preset threshold and update the marked category label of the second training text;

[0153] If the first training text data set contains a first training text whose annotated category label is inconsistent with the updated category label of the second training text, then replacing the annotated category label of the first training text with the updated category label;

[0154] The trained initial annotation model is trained using the updated first training text.

[0155] In the disclosed embodiment, when training the annotation model, the category label of the first training text is determined and updated, and the updated first training text is used to train the trained initial annotation model. Exemplarily, a second training text data set consisting of second training texts with annotated category labels is input into the trained initial annotation model for category prediction, and a third probability value that the second training text belongs to the annotated category is output, and the second classification evaluation index data of the second training text is calculated based on the third probability value and the difference between the third probability value and the second benchmark probability value, wherein the determination method and definition of the second classification evaluation index data can refer to the specific implementation of the above-mentioned first classification evaluation index data, which will not be elaborated here.

[0156] According to the second classification evaluation index data, the second training text with a value less than the preset threshold is screened out, and the label is updated and replaced. The updating and replacement method of the labeled category label of the first training text can refer to the specific implementation of updating and replacing the labeled category label of the first text. After the replacement, the updated first training text is obtained, and the updated first training text is used to train the trained initial labeling model.

[0157] In one example, the updated first training text is used to train the trained initial annotation model to obtain the initial annotation model after secondary training. For the updated first training text, the above-mentioned steps of calculating the second classification evaluation index data, updating and replacing the labels, and training the model are repeated until the second classification evaluation index data of the new first training text obtained are greater than or equal to the preset threshold, and the category label of the new first training text is determined to be the final category label.

[0158] In the disclosed embodiment, during the model training process, the label quality of the training text is optimized by the second classification evaluation index data, and the model training is performed again, which can further improve the accuracy of the model training. By optimizing the training text itself, the effect of improving the accuracy of the model training is achieved. The implementation method is simple, and both training efficiency and accuracy are taken into account. Moreover, during the model training process, the labels of the training text are updated and optimized, so that more accurate category labels of the training text can be obtained, thereby realizing category annotation of the training text, which is suitable for more application scenarios.

[0159] In one embodiment, when determining the category label, the training method of the above-mentioned annotation model can be used to obtain the first probability value of the first text during the training process, and the label is updated and replaced. The annotation model is trained again using the updated text to obtain the first probability value of the updated text, and the label is updated and replaced until the first classification evaluation index data of the first text in the text set is greater than or equal to the preset threshold, and the final category label of the first text is obtained, thereby achieving the annotation of the first text. Among them, the first classification evaluation loss data is calculated in the same way as the first classification evaluation loss data.

[0160] Figure 5 FIG. 1 is a flow chart of a method for determining a category label according to an exemplary embodiment. Figure 5 As shown, an unlabeled first text is obtained, and the first text is preliminarily labeled based on information retrieval and other methods to obtain the first text with labeled category labels; the first text with labeled category labels is used to perform K-fold cross-training of the labeling model, and after K times of training, a first probability value corresponding to the first text is obtained; based on the first probability value corresponding to the first text, first classification evaluation index data, i.e., outlier score, is calculated; outlier data in the first text is filtered out according to the outlier score, and the label of the outlier data is updated and replaced; the model training is continued using the updated first text until no outlier data can be filtered out in the first text, and the final category label is obtained, thereby achieving the labeling of the first text.

[0161] Figure 6 FIG. 1 is a flow chart of a method for training a labeling model according to an exemplary embodiment. Figure 6 As shown, in each round of iterative training, K-1 groups of data are used as training samples for training, wherein the loss function in the model training process can be obtained based on the cross entropy loss function and the contrast loss function. The remaining group of data is used as a test sample, and the test sample is input into the trained model to obtain a prediction result. The first difference and the second difference are calculated according to the prediction result to obtain a second classification evaluation loss function. The corresponding outlier data in the group of data is obtained according to the second classification evaluation loss function. After K training tests, all the outlier data in the data set are obtained.

[0162] In one example, differential texts with large differences between prediction results and category labels can also be analyzed to summarize model problems. If data problems are found during differential text analysis, the problems are fed back to the text data set to adjust and optimize the labels, and the labeling quality is continued to be checked, forming a cycle of data labeling, data quality checking, training, and analysis feedback, ultimately achieving the purpose of optimizing data labeling and obtaining a high-quality text data set that meets training standards.

[0163] Through this embodiment, the efficiency of data annotation can be effectively improved. At the same time, the solution is universal and highly scalable, and can be applied to other business scenarios that require text classification. There is no need to manually annotate all data in each new business scenario, which greatly reduces labor costs. This embodiment takes into account both data annotation speed and effect. First, the first text is quickly pre-annotated according to the second text to determine the text data set, and then the label of the first text is updated and optimized in subsequent training. Rapid annotation is performed through coarse screening, and problematic and ambiguous texts in the rapid annotation are checked through quality inspection. The annotation is adjusted in time, and the annotation problems are analyzed more comprehensively in combination with differential text analysis. The annotation results are corrected to improve the data annotation effect, and the degree of automation is higher. Through low-cost data annotation, rapid modeling is performed to conduct preliminary business exploration. If you wait for all data to be manually annotated before conducting business exploration, The search is risky, while the present embodiment can generate value at the early stage of business implementation and assist business analysis; introducing contrastive learning into the automated annotation scenario can effectively solve the text intersection and confusion problems existing in the text data set formed by search; the concept of classification evaluation index data is proposed, which can quantify the difficulty of text classification and provide a standard for screening difficult-to-classify annotated data; in traditional technologies, due to the large amount of annotated data, professional annotators are required to annotate, and professional annotators cannot quickly annotate when changing scenarios, and a certain amount of time is required for business and annotation standard training, while the present embodiment can be applied to most business scenarios with text classification requirements, and only a small amount of initial text data is required. Due to the small amount of data required, business personnel in the corresponding business scenario can be directly asked to annotate, which further reduces the workload of manual annotation and improves annotation efficiency.

[0164] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0165] Based on the same inventive concept, the embodiment of the present disclosure also provides a text data annotation device for implementing the above-mentioned method for determining category labels. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiment of the device for determining one or more category labels provided below can refer to the limitations of the method for determining category labels above, and will not be repeated here.

[0166] In an exemplary embodiment, Figure 7 As shown, a device 700 for determining a category label is provided, comprising:

[0167] A determination module 710 is used to determine a text data set, wherein the text data set includes a first text with a marked category label;

[0168] A prediction module 720, configured to predict a first probability value corresponding to the first text according to the first text;

[0169] A calculation module 730, configured to calculate first classification evaluation index data of the first text according to the difference between the first probability values ​​and the difference between the first probability value and a first reference probability value;

[0170] A screening module 740 is used to screen out the first text whose first classification evaluation index data is less than a preset threshold and update the marked category label of the first text;

[0171] The replacement module 750 is configured to replace the marked category label of the first text with the updated category label if there is a first text in the text data set whose marked category label is inconsistent with the updated category label.

[0172] In one embodiment, the computing module comprises:

[0173] A first determining submodule, configured to determine, according to a maximum probability value and a second maximum probability value in the first probability values, a first difference between the maximum probability value and the second maximum probability value, and a second difference between the maximum probability value and a first reference probability value;

[0174] A summing module is used to perform a sum operation on the first difference and the second difference to obtain first classification evaluation index data of the first text.

[0175] In one embodiment, the prediction module includes:

[0176] The prediction submodule is used to input the first text into the annotation model for category prediction and output a first probability value corresponding to the first text.

[0177] In one embodiment, the determining module includes:

[0178] An acquisition submodule, used to acquire a second text that has been annotated with an initial category label;

[0179] A second determination submodule, configured to determine a marked category label of the first text according to an association relationship between the second text and the first text;

[0180] The first obtaining module is used to obtain the text data set according to the first text and the marked category label of the first text.

[0181] In one embodiment, the second determining submodule includes:

[0182] An acquisition unit, configured to acquire keywords of the second text;

[0183] A determination unit is used to determine the initial category label of the second text as the marked category label of the first text if the keyword matches the first text and the text similarity between the second text and the first text is greater than a preset threshold.

[0184] Based on the same inventive concept, the embodiment of the present disclosure also provides a text data annotation device for implementing the above-mentioned annotation model training method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of the training device for one or more annotation models provided below can refer to the limitations of the annotation model training method above, and will not be repeated here.

[0185] In an exemplary embodiment, Figure 8 As shown, a training device 800 for a labeling model is provided, comprising:

[0186] An input module 810 is used to input a first training text data set including first training texts with annotated category labels into an initial annotation model for category prediction, and output a second probability value corresponding to the first training text;

[0187] A first determination module 820, configured to determine a classification loss function according to the marked category label and the second probability value;

[0188] A second determination module 830 is used to determine a first similarity between every two first training texts in the training text data set and a similarity label between every two first training texts in the training text data set according to the second probability value;

[0189] A third determination module 840 is used to determine a contrast loss function according to the first similarity and the similarity label;

[0190] The training module 850 is used to train the initial annotation model according to the classification loss function and the contrast loss function.

[0191] In one embodiment, the third determination module includes:

[0192] A second obtaining module is used to obtain a first component of a contrast loss function according to the first similarity, the similarity label and the number of first training text pairs consisting of every two first training texts;

[0193] A third obtaining module, configured to obtain a second component of a contrast loss function according to the first similarity, the similarity label and a preset threshold of the first similarity;

[0194] The third determination submodule is used to determine a contrast loss function according to the first component and the second component.

[0195] In one embodiment, the device further comprises:

[0196] An output submodule, used for inputting a second training text data set consisting of second training texts with annotated category labels into the trained initial annotation model for category prediction, and outputting a third probability value that the second training text belongs to the annotated category;

[0197] a calculation submodule, configured to calculate the second classification evaluation index data of the second training text according to the third probability value and the difference between the third probability value and the second reference probability value;

[0198] A screening submodule, used to screen out the second training texts whose second classification evaluation index data is less than a preset threshold and update the marked category labels of the second training texts;

[0199] A replacement submodule, configured to replace the marked category label of the first training text with the updated category label if the first training text data set contains a first training text whose marked category label is inconsistent with the updated category label of the second training text;

[0200] The training submodule is used to train the trained initial annotation model using the updated first training text.

[0201] Each module in the above-mentioned device for determining the category label and the training device for the annotation model can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0202] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the method described in this embodiment, such as a text data set. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for determining a category label and / or a method for training a labeling model is implemented.

[0203] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0204] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0205] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0206] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0207] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0208] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.

[0209] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0210] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A method for determining a category label, characterized in that: include: Determine a text data set, wherein the text data set includes a first text with annotated category labels; Predicting, according to the first text, a first probability value corresponding to the first text; Calculate the first classification evaluation index data of the first text according to the difference between the first probability values ​​and the difference between the first probability value and the first reference probability value; Filter out the first text whose first classification evaluation index data is less than a preset threshold and update the marked category label of the first text; If there is a first text in the text data set whose annotated category label is inconsistent with the updated category label, the annotated category label of the first text is replaced with the updated category label.

2. The method according to claim 1, characterized in that The first classification evaluation index data of the first text is calculated based on the difference between the first probability values ​​and the difference between the first probability value and the first reference probability value, including: Determine, according to a maximum probability value and a second maximum probability value in the first probability values, a first difference between the maximum probability value and the second maximum probability value, and a second difference between the maximum probability value and a first reference probability value; The first difference and the second difference are summed to obtain first classification evaluation index data of the first text.

3. The method according to claim 1, characterized in that The predicting, according to the first text, a first probability value corresponding to the first text includes: The first text is input into a labeling model for category prediction, and a first probability value corresponding to the first text is output.

4. The method according to claim 1, characterized in that: The determining of the text data set comprises: Get the second text with the initial category label; Determining a marked category label of the first text according to the association relationship between the second text and the first text; The text data set is obtained according to the first text and the marked category label of the first text.

5. The method according to claim 4, characterized in that The step of determining the annotated category label of the first text according to the association relationship between the second text and the first text includes: Obtaining keywords of the second text; If the keyword matches the first text, and the text similarity between the second text and the first text is greater than a preset threshold, the initial category label of the second text is determined as the marked category label of the first text.

6. A method for training a labeling model, characterized in that: include: Inputting a first training text data set including first training texts with annotated category labels into an initial annotation model for category prediction, and outputting a second probability value corresponding to the first training text; Determining a classification loss function according to the marked category label and the second probability value; Determine, according to the second probability value, a first similarity between every two first training texts in the training text data set and a similarity label between every two first training texts in the training text data set; Determining a contrast loss function according to the first similarity and the similarity label; The initial annotation model is trained according to the classification loss function and the contrast loss function.

7. The method according to claim 6, characterized in that The determining of the contrast loss function according to the first similarity and the similarity label includes: Obtaining a first component of a contrast loss function according to the first similarity, the similarity label, and the number of first training text pairs consisting of every two first training texts; Obtaining a second component of the contrast loss function according to the first similarity, the similarity label, and a preset threshold of the first similarity; A contrast loss function is determined according to the first component and the second component.

8. The method according to claim 6, characterized in that The method further comprises: Inputting a second training text data set consisting of second training texts with annotated category labels into the trained initial annotation model for category prediction, and outputting a third probability value that the second training text belongs to the annotated category; Obtain second classification evaluation index data of the second training text according to the third probability value and the difference between the third probability value and the second reference probability value; Screen out the second training text whose second classification evaluation index data is less than a preset threshold and update the marked category label of the second training text; If the first training text data set contains a first training text whose annotated category label is inconsistent with the updated category label of the second training text, then replacing the annotated category label of the first training text with the updated category label; The trained initial annotation model is trained using the updated first training text.

9. A device for determining a category label, characterized in that: The device comprises: A determination module, configured to determine a text data set, wherein the text data set includes a first text with a marked category label; A prediction module, configured to predict a first probability value corresponding to the first text according to the first text; A calculation module, used for calculating the first classification evaluation index data of the first text according to the difference between the first probability values ​​and the difference between the first probability value and the first reference probability value; A screening module, used to screen out the first text whose first classification evaluation index data is less than a preset threshold and update the marked category label of the first text; The replacement module is used to replace the marked category label of the first text with the updated category label if there is a first text in the text data set whose marked category label is inconsistent with the updated category label.

10. A training device for a labeling model, characterized in that: The device comprises: An input module, used for inputting a first training text data set consisting of first training texts with annotated category labels into an initial annotation model for category prediction, and outputting a second probability value corresponding to the first training text; A first determination module, configured to determine a classification loss function according to the marked category label and the second probability value; A second determination module is used to determine a first similarity between every two first training texts in the training text data set and a similarity label between every two first training texts in the training text data set according to the second probability value; A third determination module, configured to determine a contrast loss function according to the first similarity and the similarity label; A training module is used to train the initial annotation model according to the classification loss function and the contrast loss function.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images