Data labeling method and device, equipment, medium and program product

Abstract

The invention provides a data annotation method which can be applied to the field of big data or artificial intelligence. The method comprises the steps of obtaining business handling records in a historical time period, performing problem analysis on the business handling records, and obtaining a problem set; based on at least one question in the question set, a question statement is generated, the question statement is input into a pre-trained language large model, a prompt word set output by the language large model is obtained, and the prompt word set comprises the question and at least one corresponding emotion prompt word; inputting the cue word set as input into the large language model to obtain at least one text statement output by the large language model; and based on each text statement and each corresponding emotion prompt word, generating a labeled data set for training an emotion recognition model. The invention further provides a data annotation device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data or artificial intelligence, and more specifically to a data annotation method, device, equipment, storage medium and program product. Background Art

[0002] The large language model is a neural network with millions or even billions of parameters. By learning from massive corpora, the model has the ability to understand and generate language. However, calling a large language model requires a lot of computing resources. For tasks with high real-time requirements or relatively simple tasks, a small model is usually used. In addition, the text emotion recognition model is used to identify the emotions expressed in the text and has a wide range of applications in real-time monitoring, quality inspection and other scenarios. However, text emotion recognition tasks in different scenarios often require different labeled data to train the model. In other words, every time a new application scenario is changed, the data needs to be re-labeled and the model needs to be trained. Due to the large amount of data, manual labeling is both time-consuming and labor-intensive. Summary of the invention

[0003] In view of the above problems, embodiments of the present disclosure provide a data annotation method, apparatus, device, medium, and program product.

[0004] According to a first aspect of the present disclosure, a data labeling method is provided, the method comprising: obtaining business processing records of a historical time period, performing problem analysis on the business processing records, and obtaining a problem set; generating a problem sentence based on at least one problem in the problem set, and inputting the problem sentence into a pre-trained language model to obtain a prompt word set output by the language model, the prompt word set including the problem and at least one corresponding emotional prompt word; inputting the prompt word set as input into the language model to obtain at least one text sentence output by the language model; and generating a labeled data set for training an emotion recognition model based on each of the text sentences and the corresponding each of the emotional prompt words.

[0005] According to an embodiment of the present disclosure, the method also includes: inputting the labeled data set into the emotion recognition model and obtaining the output of the emotion recognition model; and when the number of training steps of the emotion recognition model is greater than or equal to a first specified threshold, or the difference between the output of the emotion model and the expected output is less than or equal to a second specified threshold, stopping the training to obtain the trained emotion recognition model.

[0006] According to an embodiment of the present disclosure, the method also includes: predicting unlabeled samples through the trained emotion recognition model to obtain prediction results, wherein the unlabeled samples are unlabeled samples of a manually labeled data set, and the number of the manually labeled data sets is a preset proportion of the number of labeled data sets; comparing the prediction results with the manual labeling results of the manually labeled data sets to obtain the accuracy of the prediction results; and when the accuracy is less than or equal to a third prescribed threshold or the amplitude of the change in the accuracy is greater than or equal to a fourth prescribed threshold, adjusting the parameters and structure of the emotion recognition model until the accuracy of the emotion recognition model is greater than or equal to the third prescribed threshold or the amplitude of the change in the accuracy is less than or equal to the fourth prescribed threshold.

[0007] According to an embodiment of the present disclosure, the method also includes: obtaining misrecognition data in the prediction results based on the emotion recognition model for the unlabeled samples; calling the language large model to generate similar sentences of the misrecognition data to form a supplementary data set; and combining the supplementary data set with the labeled data set, inputting them together into the emotion recognition model, and performing a retraining operation on the emotion recognition model.

[0008] According to an embodiment of the present disclosure, the business processing records of the historical time period are obtained, and problem analysis is performed on the business processing records to obtain a problem set, including: performing problem analysis on the business processing records to obtain at least one problem corresponding to the business processing records; processing each of the problems, deleting duplicate problems, and merging similar problems; and based on the type of the business processing records, integrating the processed problems by type to obtain the problem set.

[0009] According to an embodiment of the present disclosure, the step of taking the prompt word set as input and inputting it into the language big model to obtain at least one text sentence output by the language big model includes: completing first variable information in a preset input sentence based on a question in the prompt word set; completing second variable information in the input sentence based on emotional prompt words in the prompt word set; and inputting the completed input sentence into the language big model to obtain at least one text sentence output by the language big model.

[0010] According to an embodiment of the present disclosure, the method further includes: customizing third variable information of the input sentence to generate an input of the language large model, wherein the third variable information is used to characterize the number of the text sentences returned.

[0011] A second aspect of the present disclosure provides a data labeling device, which includes: a problem analysis module, which is used to obtain business processing records in a historical time period, perform problem analysis on the business processing records, and obtain a problem set; a prompt word generation module, which is used to generate a question sentence based on at least one question in the problem set, and input the question sentence into a pre-trained language model to obtain a prompt word set output by the language model, wherein the prompt word set includes the question and at least one corresponding emotional prompt word; a sentence generation module, which is used to input the prompt word set as input into the language model to obtain at least one text sentence output by the language model; and a labeling module, which is used to generate a labeled data set for training an emotion recognition model based on each of the text sentences and the corresponding each of the emotional prompt words.

[0012] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above-mentioned data labeling method.

[0013] The fourth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned data labeling method are implemented.

[0014] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data labeling method when executed by a processor.

[0015] In the embodiments of the present disclosure, the automatic generation of progressive emotion prompt words and text sentences is achieved through the input and output mechanism of the large language model, and finally a labeled data set for training the emotion recognition model is obtained, which greatly reduces the workload of manual labeling, thereby achieving the goal of reducing costs and increasing efficiency while ensuring data quality, and significantly improving the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the data annotation method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0018] Figure 2 A flowchart of a data annotation method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 3The figure shows a flowchart of outputting text sentences through a language model according to an embodiment of the present disclosure;

[0020] Figure 4 Another flow chart of the data annotation method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 5 schematically shows a structural block diagram of a data annotation device according to an embodiment of the present disclosure; and

[0022] Figure 6 The figure schematically shows a block diagram of an electronic device according to a data labeling method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0029] Figure 1 The application scenario diagram of the data annotation method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the data annotation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data annotation device provided in the embodiment of the present disclosure can generally be set in the server 105. The data annotation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data annotation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0036] Figure 2 The flowchart of the data annotation method according to the embodiment of the present disclosure is schematically shown.

[0037] like Figure 2 It is shown that the data labeling method of this embodiment includes operations S210 to S240.

[0038] In operation S210, business processing records of a historical time period are obtained, and problem analysis is performed on the business processing records to obtain a problem set.

[0039] In some exemplary embodiments, all business records within a specified time period can be extracted from the database. For example, when handling banking business, it includes but is not limited to loan business, ETC business and credit card business. For each type of business handling, problem analysis can be performed according to the specific situation. For example, when handling credit card business, card swallowing problems caused by equipment failure, improper operation or problems with the card itself, or card recognition errors caused by damaged card magnetic stripes, card reader failure or imperfect recognition algorithms, etc., thereby constructing a problem set.

[0040] In an embodiment of the present disclosure, before obtaining the business handling records of the historical time period, the user's consent or authorization is obtained. For example, before operation S210, a request to obtain the business handling records of the historical time period is issued to the user. If the user agrees or authorizes that the business handling records of the historical time period can be obtained, operation S210 is performed.

[0041] After obtaining the business processing record, execute operations S211~S212.

[0042] In operation S211, problem analysis is performed on the business processing record to obtain at least one problem corresponding to the business processing record.

[0043] Conduct in-depth analysis of the information in the business processing records, including the specific content of the request, the processing process, the result feedback, and the user feedback. Through comparative analysis, pattern recognition, and expert experience judgment, one or more issues related to the business processing request can be discovered and recorded.

[0044] In operation S212, the questions are processed, duplicate questions are deleted, and similar questions are merged.

[0045] The identified questions are further processed to remove duplicates and merge similar questions. First, advanced deduplication algorithms and techniques can be used to traverse and compare various questions, find and delete duplicate questions with exactly the same or highly similar content, so as to streamline the question set and improve the efficiency and accuracy of subsequent analysis. Secondly, for those questions that are the same or similar in essence despite different expressions, they can be merged and merged into one question, and the most representative and descriptive expressions can be retained, thereby reducing the redundancy of the question set and making the questions more focused and clear.

[0046] In operation S213, based on the type of the business processing record, the processed questions are classified and integrated to obtain the question set.

[0047] According to the different types and characteristics of banking business, the questions can be divided into different categories, such as loan business questions, ETC business questions and credit card business questions. Then, the questions under each category can be summarized and sorted to form a structured question set. This question set not only contains detailed information and descriptions of the questions, but also clarifies the category and business scope of each question.

[0048] For example, suppose that a bank's business records are being processed, which include records of loan business, ETC business, and credit card business. First, by analyzing these records, some potential problems were found. For example, in the loan business, some problems such as poor application process and long approval time were found; in the ETC business, some equipment failures and untimely handling of user complaints were found; in the credit card business, some card recognition errors and slow customer service response were found. Then, these problems are deduplicated and merged. For example, multiple records of problems with exactly the same content, such as long loan approval time, were deleted, and several ETC equipment failure problems with different expressions but the same essence were merged into one problem. Finally, the processed problems were classified and integrated according to the business type, and a structured problem set including loan business problems, ETC business problems, and credit card business problems was obtained.

[0049] It is understandable that in the process of handling problems, by deleting duplicate problems and merging similar problems, data redundancy can be effectively reduced and the simplicity and clarity of the problem set can be improved. At the same time, by classifying and integrating the processed problems, a structured and systematic problem set can be formed, which not only helps to fully understand the distribution and characteristics of problems in the process of handling various types of business, but also provides strong support for subsequent problem solving and process optimization.

[0050] In operation S220, a question sentence is generated based on at least one question in the question set, and the question sentence is input into a pre-trained language model to obtain a prompt word set output by the language model, wherein the prompt word set includes the question and at least one corresponding emotional prompt word.

[0051] According to the specific questions in the question set, they are converted into more targeted and descriptive question sentences, and then the question sentences are input into the pre-trained language model. Through the language model, emotional prompt words closely related to the questions are mined. Finally, a prompt word set containing questions and at least one corresponding emotional prompt word can be obtained. This prompt word set is used to input into the large model to generate training data.

[0052] For example, the question sentence is input into the language big model, that is, the input to the big model is "When XX, what emotions are people likely to have?" XX is the specific question corresponding to the business analyzed in the above embodiment. For example, "When the card is swallowed, what emotions are people likely to have?" The language big model returns "When the card is swallowed, people may be surprised, confused, angry, anxious, and worried." These are recorded to form a prompt word set, and the format of the prompt word set is (question; emotion 1, emotion 2, ..., emotion n). Among them, emotion 1, emotion 2, ..., emotion n are all prompt words.

[0053] In operation S230, the prompt word set is input into the language model as an input to obtain at least one text sentence output by the language model.

[0054] After obtaining the prompt word set, such as Figure 3 As shown, operations S231 to S233 are performed.

[0055] In operation S231, based on the question in the prompt word set, the first variable information in the preset input sentence is completed.

[0056] In operation S232, the second variable information in the input sentence is supplemented based on the emotion prompt words in the prompt word set.

[0057] In operation S233, the completed input sentence is input into the language model to obtain at least one text sentence output by the language model.

[0058] In some exemplary embodiments, the prompt word set output by the language large model for the first time is used as the input of the language large model again. Specifically, the business problems contained in the prompt word set and the corresponding emotional prompt words are used to construct input sentences, guiding the language large model to generate text sentences that are closely related to the business scenario and rich in emotion. In this process, firstly, based on the problems in the prompt word set, the first variable information (i.e., business problems) in the preset input sentence is supplemented. Among them, the preset input sentence is a template, which defines the basic format and framework of the input to the language large model. By filling the specific problems in the prompt word set into the template, it can be ensured that the input sentence can accurately reflect the actual problems in the business scenario. Then, based on the emotional prompt words in the prompt word set, the second variable information (i.e., emotion) in the input sentence can be supplemented. After supplementing the first variable information and the second variable information, a complete input sentence can be obtained, which not only contains business problems, but also incorporates emotional factors, providing rich context information for the language large model. Then, this supplemented input sentence can be input to the language large model, and then the model will output at least one text sentence matching the input sentence.

[0059] It is understandable that by providing the language model with inputs containing rich emotions and business information, the model can learn more features and patterns, thereby enhancing its generalization ability and robustness. At the same time, diverse inputs will inspire the language model to generate richer text sentences to increase the diversity of the training data set.

[0060] In the disclosed embodiment, the method further includes: customizing third variable information of the input sentence to generate input of the large language model, wherein the third variable information is used to characterize the number of the returned text sentences.

[0061] Furthermore, the third variable information of the input sentence can be customized, and the third variable information is used to characterize the number of returned text sentences (i.e., the value of N). By adjusting the value of N, the number of text sentences generated by the language model can be controlled, thereby generating training data sets of different sizes as needed.

[0062] It can be understood that by customizing the third variable information, the user can flexibly control the number of text sentences output by the language model, thereby ensuring the quality of training while achieving a fast and efficient training process.

[0063] Exemplarily, based on the prompt word set, an input sentence is generated and input into the big model. The format of the input sentence is: "When XX happens, people have YY emotions. What might they say? Give N sentences." XX is the problem corresponding to the business, that is, the first variable information, and YY is the emotional prompt word, that is, the second variable information. The emotional prompt words generated by the above embodiment are input in sequence. N is the number of customized text sentences to be returned, that is, the third variable information of the input sentence. For example, "When the card is swallowed, people have a worried emotion. What might they say? Give 1 sentence." The text sentence returned by the big model is "Will the money in the card be at risk?!".

[0064] In operation S240, based on each of the text sentences and the corresponding emotion prompt words, a labeled data set for training an emotion recognition model is generated.

[0065] In an embodiment of the present disclosure, a corresponding operation entry is provided for the user to choose to agree or reject the automated decision result. That is, before generating the labeled data set processing / decision for training the emotion recognition model, an instruction of the user to agree or reject the processing / decision input through the corresponding operation entry is obtained. If the user agrees to perform the processing / decision, the labeled data set processing / decision for training the emotion recognition model is generated, that is, step S240 is executed. If the user refuses to perform the processing / decision, the expert decision process is entered.

[0066] Based on the text sentences obtained from the language model and their corresponding emotion prompt words, a labeled dataset for training the emotion recognition model can be generated. The format of the dataset is (text sentence, emotion prompt word), which not only contains rich text information, but also comes with accurate emotion labels, providing a solid foundation for the training of the emotion recognition model. The format of the dataset is clear, which makes it easy for the model to quickly and accurately identify and learn the emotional features in the text during the training process.

[0067] It is understandable that in order to solve the problem of requiring a large amount of manually annotated data when changing the usage scenario of the text emotion recognition model, the disclosed embodiment provides a data annotation method, which uses a large language model to generate emotion prompt words, and uses the prompt words as input to generate text sentences, and finally generates an annotated data set. Compared with the traditional manual annotation method, this method uses a large language model for automatic annotation, and only requires a small amount of manually annotated data, which greatly reduces the annotation workload and the cost of data annotation, and finally achieves cost reduction and efficiency improvement, and improves model performance.

[0068] In an embodiment of the present disclosure, the method also includes: inputting the labeled data set into the emotion recognition model and obtaining the output of the emotion recognition model; and when the number of training steps of the emotion recognition model is greater than or equal to a first specified threshold, or the difference between the output of the emotion model and the expected output is less than or equal to a second specified threshold, stopping the training to obtain the trained emotion recognition model.

[0069] In some exemplary embodiments, the obtained annotated data set is input into an emotion recognition model (such as a BERT model) and trained. During the training process, the model will continuously learn and adjust its internal parameters to better identify and understand the emotional features in the text. In this process, in order to monitor the progress and effect of the training, the conditions for stopping the training can be set. First, when the number of training steps of the emotion recognition model reaches or exceeds the preset first specified threshold, the training process will stop automatically. Among them, the first specified threshold can be determined according to the complexity of the model and the scale of the training data to ensure that the model has enough time to learn and optimize its performance. Secondly, a stop condition based on the accuracy of the model output can also be set. When the difference between the output of the emotion recognition model and the expected output is less than or equal to the preset second specified threshold, the training process will also stop. This condition reflects the model's ability to recognize text emotions. When the model can stably and accurately recognize the emotions in the text, it is considered that it has achieved the training goal.

[0070] It can be understood that by applying the generated labeled dataset to the training of the emotion recognition model and setting clear stopping training conditions, an accurate and efficient emotion recognition model can be effectively trained.

[0071] In some exemplary embodiments, the method further includes: predicting unlabeled samples through the trained emotion recognition model to obtain prediction results, wherein the unlabeled samples are unlabeled samples of a manually labeled data set, and the number of manually labeled data sets is a preset proportion of the number of labeled data sets; comparing the prediction results with the manual labeling results of the manually labeled data set to obtain the accuracy of the prediction results; and when the accuracy is less than or equal to a third prescribed threshold or the amplitude of the change in the accuracy is greater than or equal to a fourth prescribed threshold, adjusting the parameters and structure of the emotion recognition model until the accuracy of the emotion recognition model is greater than or equal to the third prescribed threshold or the amplitude of the change in the accuracy is less than or equal to the fourth prescribed threshold.

[0072] After the emotion recognition model training is completed, the emotion recognition model can be further verified, adjusted and optimized to understand the performance of the model on unknown data, so as to determine whether it has sufficient generalization ability. In this process, a part of the manually annotated data set is selected as the verification set, and at the same time, these data are de-annotated, that is, their original emotional labels are hidden. Then, the trained emotion recognition model is used to predict the unlabeled samples, and the prediction results are recorded. Then, these prediction results are compared with the manually annotated results of the manually annotated data set one by one, and the performance of the model is quantified by calculating the accuracy. At the same time, the stop condition can also be set to further evaluate the stability and convergence of the model. First, when the accuracy of the model is less than or equal to the preset third specified threshold, it is considered that the performance of the model has not met the requirements and needs to be further adjusted and optimized. Secondly, when the change range of the model accuracy is greater than or equal to the preset fourth specified threshold, it also indicates that the performance of the model is not stable and it is necessary to continue iterative training. After multiple iterations, if the accuracy of the model still cannot reach the preset value, or the accuracy still varies greatly, the parameters and structure of the model can be adjusted, such as changing the hyperparameters of the model, increasing or decreasing the number of layers of the model, adjusting the activation function of the model, etc. Finally, when the accuracy of the model is greater than or equal to the preset third threshold, or the accuracy variation is less than or equal to the preset fourth threshold, for example, the variation is less than 0.1%, and there is no significant improvement after adding a supplementary data set, the model can be considered to have converged and reached a stable state.

[0073] It should be noted that the number of labeled samples is controlled within a small preset ratio of the number of samples in the training set (i.e., labeled data set), such as 1 / 10, to reduce the cost and time consumption of manual labeling, making the entire data processing process more efficient. In actual operations, labeling work usually requires a lot of manpower and time, especially when processing large-scale data sets. By reducing the number of samples that need to be labeled, the burden on labelers can be significantly reduced, while shortening the model training cycle.

[0074] It is understandable that by comparing the prediction results with the true labels, the performance of the emotion recognition model on unknown data can be intuitively evaluated, so that the model can better adapt to the unknown data and improve its generalization ability.

[0075] In some exemplary embodiments, the method further includes: obtaining misrecognition data in the prediction results of the emotion recognition model for the unlabeled samples; calling the large language model to generate similar sentences of the misrecognition data to form a supplementary data set; and combining the supplementary data set with the labeled data set, inputting them together into the emotion recognition model, and performing a retraining operation on the emotion recognition model.

[0076] After predicting the unlabeled samples by the emotion recognition model, the misidentified data can be identified based on the prediction results. The misidentified data is the sample whose prediction results of the model do not match the real emotion label. After obtaining the misidentified data, a large language model can be called to generate similar sentences of these data, that is, to generate sentences that are similar to the misidentified sentences in terms of grammar, semantics and style, so as to build a more comprehensive and rich supplementary data set. Then, the supplementary data set is combined with the original labeled data set to form a larger training data set, or the supplementary data set is input into the trained emotion recognition model as a new training data set to retrain the emotion recognition model to improve the accuracy and generalization ability of the model by introducing more diverse data and features. Similarly, an end condition can also be set, such as the accuracy of the model on the validation set reaches a preset threshold or the change in accuracy is less than a certain threshold. Before the condition is met, the above steps can be repeated, that is, identifying misidentified data, calling a large language model to generate similar sentences, building a supplementary data set, and retraining with the data set until the end condition is met, thereby outputting the final emotion recognition model.

[0077] It is understandable that by identifying misidentified data and generating similar sentences in a targeted manner, and constructing a supplementary data set for retraining, it is possible to more accurately learn the parts of the emotion recognition model that are prone to errors in prediction, thereby effectively improving the performance of the model and the overall recognition accuracy. In addition, it can greatly enrich the diversity of training data, help the model better generalize to unseen data, and improve the robustness of the emotion recognition model.

[0078] Figure 4 Another flowchart of the data annotation method according to an embodiment of the present disclosure is schematically shown.

[0079] like Figure 4 As shown, the data annotation method provided by the embodiment of the present disclosure first performs problem analysis on the business processing records. Then, a series of prompt words (emotional prompt words) are generated through the language big model, and based on these prompt words, the language big model is used to generate text sentences to form a training set (annotated data set). Then, the model is trained based on the training set. After the training is completed, the model is used to predict the manually annotated samples, and the prediction results are compared with the manually annotated results to perform accuracy analysis. When the accuracy rate does not reach the preset threshold, a supplementary data set can be generated for the misidentified data in the prediction results, and the training set is updated based on the supplementary data set, and the model is iteratively trained. When the accuracy rate reaches the preset threshold, the final emotion recognition model is output.

[0080] Figure 5 The structural block diagram of the data annotation device according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 5 As shown, the data annotation device 500 according to this embodiment includes a question analysis module 510 , a prompt word generation module 520 , a sentence generation module 530 and an annotation module 540 .

[0082] The problem analysis module 510 is used to obtain business handling records in a historical time period, perform problem analysis on the business handling records, and obtain a problem set. In one embodiment, the problem analysis module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0083] The prompt word generation module 520 is used to generate a question sentence based on at least one question in the question set, and input the question sentence into a pre-trained language model to obtain a prompt word set output by the language model, wherein the prompt word set includes the question and at least one corresponding emotional prompt word. In one embodiment, the prompt word generation module 520 can be used to perform the operation S220 described above, which will not be described in detail here.

[0084] The sentence generation module 530 is used to input the prompt word set as input into the language model to obtain at least one text sentence output by the language model. In one embodiment, the sentence generation module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0085] The annotation module 540 is used to generate an annotated data set for training the emotion recognition model based on each of the text sentences and the corresponding emotion prompt words. In one embodiment, the annotation module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0086] In an embodiment of the present disclosure, the labeling module 540 can also be used to: input the labeled data set into the emotion recognition model and obtain the output of the emotion recognition model; and when the number of training steps of the emotion recognition model is greater than or equal to a first specified threshold, or the difference between the output of the emotion model and the expected output is less than or equal to a second specified threshold, stop training to obtain the trained emotion recognition model.

[0087] In an embodiment of the present disclosure, the labeling module 540 can also be used to: predict unlabeled samples through the trained emotion recognition model to obtain prediction results, wherein the unlabeled samples are unlabeled samples of a manually labeled data set, and the number of manually labeled data sets is a preset proportion of the number of labeled data sets; compare the prediction results with the manual labeling results of the manually labeled data set to obtain the accuracy of the prediction results; and when the accuracy is less than or equal to a third prescribed threshold or the amplitude of the change in the accuracy is greater than or equal to a fourth prescribed threshold, adjust the parameters and structure of the emotion recognition model until the accuracy of the emotion recognition model is greater than or equal to the third prescribed threshold or the amplitude of the change in the accuracy is less than or equal to the fourth prescribed threshold.

[0088] In an embodiment of the present disclosure, the labeling module 540 can also be used to: obtain misrecognition data in the prediction results based on the prediction results of the emotion recognition model for the unlabeled samples; call the language model to generate similar sentences of the misrecognition data to form a supplementary data set; and combine the supplementary data set with the labeled data set, input them together into the emotion recognition model, and perform a retraining operation on the emotion recognition model.

[0089] In an embodiment of the present disclosure, the problem analysis module 510 is specifically used to: perform problem analysis on the business processing record to obtain at least one problem corresponding to the business processing record; process each of the problems, delete duplicate problems, and merge similar problems; and based on the type of the business processing record, integrate the processed problems by type to obtain the problem set.

[0090] In an embodiment of the present disclosure, the sentence generation module 530 is specifically used to: complete the first variable information in a preset input sentence based on the questions in the prompt word set; complete the second variable information in the input sentence based on the emotional prompt words in the prompt word set; and input the completed input sentence into the language model to obtain at least one text sentence output by the language model.

[0091] In an embodiment of the present disclosure, the sentence generation module 530 may also be used to: customize the third variable information of the input sentence to generate the input of the language large model, wherein the third variable information is used to characterize the number of the returned text sentences.

[0092] According to an embodiment of the present disclosure, any multiple modules of the problem analysis module 510, the prompt word generation module 520, the sentence generation module 530 and the annotation module 540 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the problem analysis module 510, the prompt word generation module 520, the sentence generation module 530 and the annotation module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the question analysis module 510 , the prompt word generation module 520 , the sentence generation module 530 and the annotation module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0093] Figure 6 A block diagram of an electronic device according to a data labeling method according to an embodiment of the present disclosure is schematically shown.

[0094] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0095] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0096] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0097] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0098] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0099] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data annotation method provided by the embodiment of the present disclosure.

[0100] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0101] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0102] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0103] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0104] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0106] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data labeling method, characterized in that: The method comprises: Obtain business handling records for a historical period, perform problem analysis on the business handling records, and obtain a problem set; Based on at least one question in the question set, generate a question sentence, input the question sentence into a pre-trained language model, and obtain a prompt word set output by the language model, wherein the prompt word set includes the question and at least one corresponding emotional prompt word; Taking the prompt word set as input, inputting it into the language model, and obtaining at least one text sentence output by the language model; and Based on each of the text sentences and the corresponding emotion prompt words, a labeled data set for training an emotion recognition model is generated.

2. The method according to claim 1, characterized in that: The method further comprises: Inputting the labeled data set into the emotion recognition model and obtaining the output of the emotion recognition model; and When the number of training steps of the emotion recognition model is greater than or equal to a first specified threshold, or the difference between the output of the emotion model and the expected output is less than or equal to a second specified threshold, the training is stopped to obtain the trained emotion recognition model.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Predicting the unlabeled samples by the trained emotion recognition model to obtain prediction results, wherein the unlabeled samples are unlabeled samples of the manually labeled data set, and the number of the manually labeled data sets is a preset proportion of the number of the labeled data sets; Comparing the prediction result with the manual annotation result of the manual annotation data set to obtain the accuracy of the prediction result; and When the accuracy rate is less than or equal to a third prescribed threshold or the amplitude of the change in the accuracy rate is greater than or equal to a fourth prescribed threshold, adjust the parameters and structure of the emotion recognition model until the accuracy rate of the emotion recognition model is greater than or equal to the third prescribed threshold or the amplitude of the change in the accuracy rate is less than or equal to the fourth prescribed threshold.

4. The method according to claim 3, characterized in that: The method further comprises: Based on the prediction result of the emotion recognition model for the unlabeled sample, obtaining misrecognition data in the prediction result; Calling the language big model to generate similar sentences of the misrecognized data to form a supplementary data set; and The supplementary data set is combined with the labeled data set, and the two are input into the emotion recognition model to perform a retraining operation on the emotion recognition model.

5. The method according to claim 1, characterized in that: The obtaining of business handling records for a historical time period, performing problem analysis on the business handling records, and obtaining a problem set includes: Performing problem analysis on the business handling record to obtain at least one problem corresponding to the business handling record; Process each of the issues, remove duplicates, and merge similar issues; and Based on the type of the business handling record, the processed problems are classified and integrated to obtain the problem set.

6. The method according to claim 1 or 5, characterized in that: The step of inputting the prompt word set as input into the language model to obtain at least one text sentence output by the language model includes: Based on the questions in the prompt word set, the first variable information in the preset input sentence is completed; Based on the emotional prompt words in the prompt word set, the second variable information in the input sentence is completed; and The completed input sentence is input into the large language model to obtain at least one text sentence output by the large language model.

7. The method according to claim 6, characterized in that The method further comprises: The third variable information of the input sentence is customized to generate an input of the large language model, wherein the third variable information is used to characterize the number of the returned text sentences.

8. A data labeling device, characterized in that: The device comprises: A problem analysis module is used to obtain business handling records in a historical period, perform problem analysis on the business handling records, and obtain a problem set; A prompt word generation module, used to generate a question sentence based on at least one question in the question set, and input the question sentence into a pre-trained language model to obtain a prompt word set output by the language model, wherein the prompt word set includes the question and at least one corresponding emotional prompt word; a sentence generation module, configured to input the prompt word set as input into the language macro model, and obtain at least one text sentence output by the language macro model; and The annotation module is used to generate an annotated data set for training the emotion recognition model based on each of the text sentences and the corresponding each of the emotion prompt words.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images