Business text labeling method and device, electronic equipment and storage medium
By combining preset prompt words and target language model, automatically labeling and training the classification model, the problems of high training costs of traditional classification models and poor accuracy of language model labeling are solved, and efficient and accurate business text labeling is achieved.
Patent Information
- Application Number
- CN202411704772.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-01-14
AI Technical Summary
Traditional classification models require a lot of manual annotation during training, which consumes high costs and time. The language model does not understand business proprietary nouns and classification standards, resulting in poor labeling accuracy.
Through the pre-generated target language model and preset prompt words, the classification label of business text is automatically determined, the classification model is trained, and the results of the language model and classification model are combined to reduce manual annotation and improve label accuracy.
It reduces the labor cost and time of training classification models, improves the accuracy of target classification labels, and solves the problem of poor accuracy of individual labeling of language models.
Smart Images
Figure CN119311879A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of natural language processing technology, and in particular to a business text annotation method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of artificial intelligence technology, text classification technology has been widely used in various fields.
[0003] In the prior art, a traditional classification model is usually used to annotate the type of business text. Alternatively, a language model is used to classify and label business text. However, in the process of implementing the present invention, it is found that the prior art has at least the following technical problems: the traditional classification model requires manual labeling of a large number of sample business texts during training, and the traditional classification model is obtained based on the labeled sample business text training, which consumes a lot of labor costs and time costs. The language model does not understand the proper nouns in the business and the business classification standards, resulting in poor accuracy of the labeling results. Summary of the invention
[0004] The embodiments of the present invention provide a business text annotation method, device, electronic device and storage medium to reduce the labor cost and time cost of training a classification model and achieve the purpose of improving the accuracy of the determined target classification label.
[0005] According to one aspect of the present invention, a business text annotation method is provided, comprising:
[0006] Acquire a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text;
[0007] Determine a first classification label corresponding to the current business text based on a pre-generated target language model and a preset prompt word, train at least one classification model based on the first classification label and the current business text, and determine a second classification label for the current business text based on the classification model;
[0008] A target classification label of the current service text is determined based on the first classification label and the second classification label of the current service text.
[0009] According to another aspect of the present invention, a business text annotation device is provided, comprising:
[0010] A data set acquisition module, used to acquire a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text;
[0011] A model training module, used to determine a first classification label corresponding to the current business text based on a pre-generated target language model and a preset prompt word, train at least one classification model based on the first classification label and the current business text, and determine a second classification label of the current business text based on the classification model;
[0012] The label determination module is used to determine a target classification label of the current business text based on the first classification label and the second classification label of the current business text.
[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0014] at least one processor; and
[0015] a memory communicatively connected to the at least one processor; wherein,
[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the business text annotation method described in any embodiment of the present invention.
[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the business text annotation method described in any embodiment of the present invention when executed.
[0018] The technical solution of the embodiment of the present invention obtains a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text; based on a pre-generated target language model and a preset prompt word, a first classification label corresponding to the current business text is determined, and at least one classification model is trained based on the first classification label and the current business text, thereby determining the first classification label through the target language model and the preset prompt word, thereby eliminating the need for manual labeling, reducing the labor cost and time cost of training the classification model; and determining the second classification label of the current business text based on the classification model; based on the first classification label and the second classification label of the current business text, determining the target classification label of the current business text, so that when determining the target classification label, the first classification label obtained by the target language model and the second classification label obtained by the classification model are combined at the same time, solving the problem of poor labeling accuracy caused by determining the target classification label only through the target language model, and achieving the effect of improving the accuracy of the determined target classification label.
[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 is a flowchart of a business text annotation method provided according to an embodiment of the present invention;
[0022] Figure 2 is a flowchart of another business text annotation method provided according to an embodiment of the present invention;
[0023] Figure 3 is a marking flow chart provided according to an embodiment of the present invention;
[0024] Figure 4 is a structural schematic diagram of a business text annotation device provided according to an embodiment of the present invention;
[0025] Figure 5 It is a structural schematic diagram of an electronic device for implementing the business text annotation method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "etc." and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.
[0029] Figure 1 This is a flow chart of a business text annotation method provided according to an embodiment of the present invention. This embodiment is applicable to the case of business text annotation classification, and the method can be executed by a business text annotation device, which can be implemented in the form of hardware and / or software.
[0030] like Figure 1 As shown, the method of this embodiment may specifically include:
[0031] S110 . Acquire a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text.
[0032] In this embodiment, the target business system may be a game system and / or an e-commerce platform. The current business text may be a conversation text or a note text sent by the user end to the e-commerce platform. The current business text may also be a text such as a game name or game jargon in the game system. In order to provide better services to users, it is necessary to establish a classification label for the current business text. Exemplarily, the classification label may include a violation label and / or an emotion classification label. The emotion classification label may include at least one of a positive label, a negative label, a happy label, a sad label, a calm label, and an angry label.
[0033] In a specific implementation, the current business texts that need to determine the classification labels in the target business system can be periodically acquired according to a preset acquisition time to form the current business data set; or, when the preset acquisition quantity is reached, the preset acquisition quantity of current business texts that have not determined the classification labels in the target business system is acquired to form the current business data set.
[0034] S120, based on a pre-generated target language model and preset prompt words, determine a first classification label corresponding to the current business text, based on the first classification label and the current business text, train at least one classification model, and determine a second classification label for the current business text based on the classification model.
[0035] The target language model can be a large model that determines the classification labels of business texts. A large model, also known as a large language model or a basic model, refers to a machine learning model with a large parameter scale and complex computing structure, such as the GPT model, the Gemini model, and the Yanxi large model.
[0036] In this embodiment, preset prompt words can be set in advance for the target language model. Exemplarily, preset prompt words can be generated based on the classification standards of business data written by business personnel, the commonly used term explanations collected in the business, and examples of business text classification. The first classification label corresponding to each current business text is determined by the preset prompt words and the target language model. A training data set composed of the current business text and the first classification label is used to train at least one classification model. Exemplarily, the classification model may include a convolutional neural network model (TextCnn) for text classification and a TinyBert model. The TinyBert model is a lightweight model that compresses the language representation model through knowledge distillation technology. The classification model has a small number of parameters, low training and reasoning resource requirements, and a fast reasoning speed.
[0037] In this embodiment, when a classification model is obtained through training, each current business text may be input into the trained classification model, and the second classification label of the current business text may be determined based on the result output by the classification model.
[0038] In order to improve the accuracy of the determined second classification label, multiple classification models can be trained, and the second classification label can be determined based on the multiple classification models. Optionally, based on the first classification label and the current business text, at least one classification model is trained, and the specific implementation method of determining the second classification label of the current business text based on the classification model can be: based on the current business text and the first classification label, a first training data set and a second training data set are generated; based on the first training data set, a first classification model is trained to obtain a first classification model, and based on the second training data set, a second classification model is trained to obtain a second classification model; based on the first classification model, the second classification label of the current business text in the second training data set is determined, and based on the second classification model, the second classification label of the current business text in the first training data set is determined.
[0039] Specifically, all current business texts can be divided into two parts, one part of the current business texts and the corresponding first classification labels form a first training data set, and the other part of the current business texts and the corresponding first classification labels form a second training data set. Exemplarily, the two parts of the current business texts have the same number.
[0040] In a specific implementation, the first classification model to be trained can be trained based on the first training data set to obtain the first classification model. The second classification model to be trained can be trained based on the second training data set to obtain the second classification model. It should be noted that the first classification model to be trained and the second classification model to be trained can be the same model or different models. Exemplarily, the first classification model to be trained is the TextCnn model, and the second classification model to be trained is the TinyBert model. In order to improve the accuracy of determining the second classification label, the second classification label can be determined by cross-marking. Specifically, the current business texts in the first training data set can be respectively input into the second classification model, and the labels output by the second classification model are used as the second classification labels of the current business texts in the first training data set. The current business texts in the second data set are respectively input into the first classification model, and the labels output by the first classification model are used as the second classification labels of the current business texts in the second training data set.
[0041] This embodiment trains two classification models and uses a cross-labeling method to determine the second classification label of the current business text, which is beneficial to improving the accuracy of the determined second classification label.
[0042] S130: Determine a target classification label for the current business text based on the first classification label and the second classification label for the current business text.
[0043] In this embodiment, for each current business text, if the first classification label and the second classification label of the current business text are consistent, the first classification label or the second classification label can be used as the target classification label of the current business text. For the current business text with inconsistent first classification label and second classification label, the current business text with inconsistent labels can be sent to the business terminal to determine the target classification label of the current business text through manual annotation by business personnel.
[0044] The technical solution of the embodiment of the present invention obtains a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text; based on a pre-generated target language model and a preset prompt word, a first classification label corresponding to the current business text is determined, and at least one classification model is trained based on the first classification label and the current business text, thereby determining the first classification label through the target language model and the preset prompt word, thereby eliminating the need for manual labeling, reducing the labor cost and time cost of training the classification model; and determining the second classification label of the current business text based on the classification model; based on the first classification label and the second classification label of the current business text, determining the target classification label of the current business text, so that when determining the target classification label, the first classification label obtained by the target language model and the second classification label obtained by the classification model are combined at the same time, solving the problem of poor labeling accuracy caused by determining the target classification label only through the target language model, and achieving the effect of improving the accuracy of the determined target classification label.
[0045] Figure 2 It is a flowchart of another business text annotation method provided according to an embodiment of the present invention. Based on the above embodiment, this embodiment further includes: dividing the current business data set into a test data set and a pre-labeled data set before training the classification model; determining the true classification label of the test business text in the test data set based on the first labeling information corresponding to the test data set sent by the business terminal; updating the preset prompt words based on the test business text and the true classification label; and training at least one classification model based on the updated preset prompt words. Among them, the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:
[0046] S210: Acquire a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text.
[0047] S220: Based on the pre-generated target language model and the preset prompt words, determine a first classification label corresponding to the current business text, and divide the current business data set into a test data set and a pre-labeled data set.
[0048] In this embodiment, for each first classification label, at least one current business text corresponding to the first classification label can be selected to form a test data set. The current business texts in the current business data set excluding the test data set form a pre-labeled data set. The data volume of the test data set is smaller than the data volume of the pre-labeled data set.
[0049] In order to enable the test data set to cover the current business texts with different characteristics in the current business data set as comprehensively as possible, the current business data set can be divided into a test data set and a pre-labeled data set in the following way: for each type of first classification label, the current business texts corresponding to the first classification label are clustered to obtain at least one clustered data subset, and at least one current business text is extracted from each clustered data subset as a test business text; the test business texts form a test data set, and the remaining business texts in the current business data set excluding the test data set form a pre-labeled data set; wherein the number of remaining business texts is greater than a first preset threshold.
[0050] Specifically, for each type of first classification label, the current business text corresponding to the first classification label is subjected to text vector encoding processing to obtain a current business vector corresponding to the current business text. Exemplarily, a semantic model for converting text into a semantic space vector can be pre-trained, the current business text is input into the semantic model, and the output vector of the semantic model is used as the current business vector corresponding to the current business text. For each type of first classification label, the current business vector corresponding to the first classification label is clustered to obtain at least one clustered data subset. Optionally, the clustering process can be performed by at least one of the K-means algorithm, the hierarchical clustering algorithm, and DBSCAN (Density-Based Spatial Clustering of Applications with Noise, a density-based clustering method with noise).
[0051] After clustering the current business vectors corresponding to each first classification label, at least one current business vector can be extracted from each cluster data subset, and the current business text corresponding to the extracted current business vector is used as the test business text, and the test data set is composed of each test business text. It should be noted that the test data set is a data set for updating preset prompt words. The remaining business texts of the current business data set minus the test data set constitute a pre-labeled data set. The pre-labeled data set is a data set for training a classification model. In order to ensure the training effect of the classification model, the number of remaining business texts can be set to be greater than the first preset threshold. Those skilled in the art can determine the specific value of the first preset threshold according to the actual training effect of the classification model, and this embodiment does not limit this.
[0052] This embodiment uses clustering processing to determine the test data set, so that the test data set can fully reflect the text features in the current business data set, thereby facilitating improving the comprehensiveness of the updated preset prompt words.
[0053] S230: Determine a true classification label of the test service text in the test data set based on the first labeling information corresponding to the test data set sent by the service terminal.
[0054] It should be noted that since the comprehensiveness and accuracy of the preset prompt words affect the labeling accuracy of the target language model, the preset prompt words need to be updated through the test data set, which means that the accuracy requirements for the labels of the test business texts in the test data set are relatively high. Therefore, a first labeling request corresponding to the test data set can be generated, and the first labeling request can be sent to the business terminal so that the business personnel can manually label the test business texts in the test data set. The business terminal generates the first labeling information based on the manual labeling results, and feeds the first labeling information back to the business text annotation system. The first labeling information includes the real classification label of the test business text.
[0055] In this embodiment, after receiving the first labeling information sent by the service terminal, the first labeling information may be parsed to determine the real classification label corresponding to each test service text in the test data set.
[0056] S240. Based on the test business text and the real classification label, update the preset prompt word; based on the updated preset prompt word and the target language model, update the first classification label corresponding to the pre-labeled business text in the pre-labeled data set; based on the pre-labeled business text and the updated first classification label, train at least one classification model, and determine the second classification label of the pre-labeled business text based on the classification model.
[0057] In order to improve the classification accuracy of the target language model, the preset prompt words can be updated based on the test business text and the real classification label. Exemplarily, typical examples in the preset prompt words can be replaced or added through the test business text and the real classification label to complete the update of the preset prompt words. Further, based on the updated preset prompt words and the target language model, the latest classification label of each pre-tagged business text in the pre-tagged data set can be determined, and the first classification label can be updated to the latest classification label, thereby improving the accuracy of the first classification label of the preset tagged business text based on the updated preset prompt words.
[0058] Furthermore, at least one classification model may be trained using the pre-labeled business texts and the updated first classification labels. Each pre-labeled business text is input into the trained classification model, and the label output by the classification model is used as the second classification label of the pre-labeled business text.
[0059] This embodiment updates the preset prompt words through the test business text, and based on the updated preset prompt words, makes the first classification label more accurate, thereby improving the training accuracy of the classification model.
[0060] S250: Determine a target classification label for the current business text based on the first classification label and the second classification label of the pre-labeled business text.
[0061] It should be noted that, according to the above description, the current business text is divided into a test business text and a pre-labeled business text. When determining the target classification label of the current business text, it can also be determined from the two aspects of the test business text and the pre-labeled business text. Specifically, for the test business text, the real classification label of the test business text can be determined as the target classification label. For the pre-labeled business text, the updated first classification label and the second classification label can be compared. When the first classification label corresponding to the pre-labeled business text matches the second classification label, the first classification label or the second classification label is used as the target classification label.
[0062] Alternatively, the implementation method of determining the target classification label of the current business text based on the first classification label and the second classification label of the current business text may also be: for each pre-labeled business text in the pre-labeled data set, when the first classification label corresponding to the pre-labeled business text does not match the second classification label, the pre-labeled business text is treated as the first difficult business text; a first labeling instruction is generated based on the first difficult business text, and the first labeling instruction is sent to the business terminal to determine the target classification label corresponding to the first difficult business text based on the first labeling information corresponding to the first difficult business text fed back by the business terminal.
[0063] In a specific implementation, for each pre-labeled business text in the pre-labeled data set, if the first classification label and the second classification label corresponding to the pre-labeled business text do not match, the pre-labeled business text can be used as the first difficult business text. The first classification label and the second classification label of each pre-labeled business text are compared to screen out the first difficult business text in the pre-labeled data set.
[0064] In order to improve the accuracy of the determined target classification label, a first labeling instruction may be generated based on the first difficult business text, and the first labeling instruction may be sent to the business terminal. The first labeling instruction is an instruction for instructing the business terminal to label at least one difficult business text. After the business terminal receives the first labeling instruction, the difficult business text may be manually labeled by business personnel, and the business terminal generates first labeling information based on the labeling result of the difficult business text, and feeds the first labeling information back to the business text annotation system, so that the classification label corresponding to the first labeling information is used as the target classification label corresponding to the first difficult business text.
[0065] This embodiment obtains a manually marked classification label by sending a first marking instruction corresponding to the first difficult business text to the business terminal, and uses the manually marked classification label as the target classification label of the first difficult business text, thereby improving the accuracy of determining the target classification label; and, the first difficult business text is screened out from the pre-marked business texts, ensuring the marking accuracy while minimizing the workload of manual marking.
[0066] In order to avoid the excessive manual labeling workload caused by the first difficult business text, resulting in high labeling costs, optionally, before generating the first labeling instruction based on the first difficult business text, it also includes: determining whether the number of the first difficult business texts is greater than a second preset threshold; if it is greater than the second preset threshold, updating the test data set and the pre-labeling data set based on the first difficult business text, and repeating the operations of updating the preset prompt words, updating the first classification label, and training the classification model based on the updated test data set and the pre-labeling data set, updating the second classification label based on the trained classification model, and updating the first difficult business text based on the updated first classification label and the second classification label; generating the first labeling instruction based on the first difficult business text, including: generating the first labeling instruction based on the updated first difficult business text when the updated first difficult business text meets the preset conditions; wherein the preset conditions include that the number of the updated first difficult business texts is less than or equal to the second preset threshold, and greater than the third preset threshold.
[0067] The second preset threshold may be determined according to the marking workload of each business text. For example, the second preset threshold may be half of the total amount of pre-marked business texts.
[0068] In a specific implementation, if the number of first difficult business texts is greater than the second preset threshold, indicating that the labeling workload of the first difficult business texts is large, the test data set and the pre-labeled data set can be updated based on the first difficult business texts. Specifically, the second classification label corresponding to the first difficult business text can be used as a difficult classification label, and for each difficult classification label, the first difficult business text corresponding to the difficult classification label is subjected to text vector encoding processing, and the vector after the text vector encoding processing is clustered to obtain at least one clustered difficult data set, and at least one first difficult business text is selected from each clustered difficult data set as a test business text, and added to the test data set to complete the update of the test data set; the remaining first difficult business texts are used as pre-labeled business texts to form a new pre-labeled data set to achieve the update of the pre-labeled data set.
[0069] In order to reduce the workload generated by manual labeling, the newly added test business texts in the test data set can be sent to the business terminal for manual labeling, and the test data set can be updated based on the manually labeled classification labels; based on the updated test data set and the pre-labeled data set, the above model training process is repeated. Specifically, based on the updated test data set, the operation of updating the preset prompt words is performed, based on the updated preset prompt words and the target language model, the first current label of each pre-labeled business text in the updated pre-labeled data set is determined, and the first classification label is updated based on the first current label. Based on the updated first classification label, the model training operation is performed again.
[0070] Exemplarily, the pre-labeled business texts of the pre-labeled data set are divided into two parts, one part of the pre-labeled business texts and the updated first classification labels corresponding to the pre-labeled business texts are used to train the first classification model; the other part of the pre-labeled business texts and the updated first classification labels corresponding to the pre-labeled business texts are used to train the second classification model. The second current label of each pre-labeled business text is determined based on the first classification model and / or the second classification model, and the second classification label is updated based on the second current label.
[0071] Furthermore, according to the above method of determining difficult business texts, the first difficult business texts are screened out from the current pre-marked business texts, and it is determined whether the number of the current first difficult business texts is greater than the second preset threshold, and if it is greater than the second preset threshold, the above operation is repeated until the preset condition is met. For example, the preset condition includes that the number of the updated first difficult business texts is less than or equal to the second preset threshold, and greater than the third preset threshold; the third preset threshold may be 0.
[0072] In this embodiment, when the updated first difficult business text meets the preset conditions, a first labeling instruction is generated based on the updated first difficult business text, so as to obtain the first manual classification label of the updated first difficult business text through the first labeling instruction, and use the first manual classification label as the target classification label of the first difficult business text.
[0073] In addition, when the first difficult business text meets the preset conditions, each pre-labeled business text except the first difficult business text can be re-input into the current first classification model / or second classification model to determine the fourth classification label of the pre-labeled business text, and compare the fourth classification label with the second classification label. If the fourth classification label matches the second classification label, the fourth classification label or the second classification label can be used as the target classification label of the pre-labeled business text. If there is a pre-labeled business text whose fourth classification label does not match the second classification label, the pre-labeled business text can be used as the newly added first difficult business text, and the above operations such as determining whether the number of first difficult business texts is greater than the second preset threshold, updating the test data set and the pre-labeled data set are repeated until the preset stop repetition condition is met, and the repetition operation is terminated. Among them, the stop repetition condition includes that the number of repetitions is greater than the preset number threshold; or, the number of pre-labeled business texts whose fourth classification label matches the second classification label is equal to 0.
[0074] In this embodiment, different methods of determining target classification labels of the first difficult business texts are set according to the number of the first difficult business texts, so as to avoid the problem of large manual labeling workload caused by the first difficult business texts, which is beneficial to improving labeling efficiency and reducing labeling costs.
[0075] In order to simplify the operation process, after determining the true classification label of the test business text in the test data set, it also includes: updating the target language model based on the test business text and the true classification label; determining the third classification label corresponding to the pre-labeled business text in the pre-labeled data set based on the updated target language model; determining the second difficult business text in the pre-labeled business text based on the first classification label and the third classification label of the pre-labeled business text; generating a second labeling instruction based on the second difficult business text, and sending the second labeling instruction to the business terminal to determine the target classification label corresponding to the second difficult business text based on the second labeling information corresponding to the second difficult business text fed back by the business terminal.
[0076] In a specific implementation, the language model can be retrained based on the real classification label and the test business text, and the retrained language model can be updated to the target language model. The pre-labeled business text in the pre-labeled data set is input into the updated target language model, and the third classification label is determined based on the output result of the target language model. By comparing the first classification label and the third classification label corresponding to each pre-labeled business text, the second difficult business text whose first classification label and the third classification label do not match is determined in the pre-labeled business text.
[0077] Furthermore, when the number of the second difficult business texts is less than the second preset threshold, a second labeling instruction can be generated based on the second difficult business texts, and the second labeling instruction can be sent to the business terminal, and the second labeling information corresponding to the second difficult business text fed back by the business terminal is parsed, and the second manual classification label obtained by the parsing is used as the target classification label of the second difficult business text. When the number of the second difficult business texts is greater than or equal to the second preset threshold, the pre-labeling data set and the test data set can be updated based on the second difficult business texts, and the above operations of determining the true classification label of the test business text, updating the target language model, determining the third classification label, and determining the second difficult business text are repeated until the number of the updated second difficult business texts is less than or equal to the second preset threshold.
[0078] In this embodiment, by updating the target language model, a method for determining the second difficult business text is provided, without the need to train the first classification model and the second classification model, thereby simplifying the process of determining the target classification label and improving the efficiency of determining the target classification label.
[0079] The above describes in detail the embodiments corresponding to the business text annotation method. In order to make the technical solution of the method more clear to those skilled in the art, a specific application scenario is given below. Optionally, the target business system may be an e-commerce platform, and the current business text may be a conversation text generated in the e-commerce platform. Figure 3 As shown, the process of determining the target classification label of the current business text includes the following steps:
[0080] 1. Collect unlabeled current business texts, and determine the first classification label of the current business text based on the preset prompt words and the target language model. 2. Vectorize the current business text, cluster the obtained vectors, and generate a test data set and a pre-labeled data set based on the clustering results. 3. Obtain the true classification label of the test business text in the test data set, and update the preset prompt words based on the true classification label. 4. Re-label the pre-labeled data set based on the updated preset prompt words and the target language model, and update the first classification label based on the re-labeled label. Based on the updated first classification label, train two classification models respectively. 5. Determine the second classification label of the pre-labeled business text through the two classification models; the pre-labeled business text whose first classification label and the second classification label do not match is regarded as the first difficult business text. 6. Determine whether the number of first difficult business texts is greater than the second preset threshold. 7. If it is greater, update it to the current business text based on the first difficult business text, and repeat steps 2-5. 8. If it is less than or equal to the second preset threshold, generate a first labeling instruction based on the first difficult business text, and send the first labeling instruction to the business terminal to obtain the target classification label of the first difficult business text.
[0081] This embodiment combines the target language model and two classification models to label the current business text. The target language model has strong understanding ability, and the classification model has fast training iteration and online reasoning speed. Combined with a small amount of manually labeled data, a low-cost and high-quality text labeling method is provided, thereby minimizing labor costs and time costs. At the same time, during the labeling process, by updating the preset prompt words, business knowledge is continuously integrated into the target language model, thereby improving the target language model's understanding ability and classification accuracy of the current business text; the classification errors of the target language model are discovered and corrected in a low-cost iteration process.
[0082] Figure 4 1 is a schematic diagram of the structure of a business text annotation device provided according to an embodiment of the present invention, and the device is used to execute the business text annotation method provided by any of the above embodiments. The device and the business text annotation method of the above embodiments belong to the same inventive concept, and the details not described in detail in the embodiment of the business text annotation device can refer to the embodiment of the above business text annotation method. Figure 4 As shown, the device comprises:
[0083] The data set acquisition module 10 is used to acquire the current business data set to be labeled from the target business system; wherein the current business data set includes at least one unlabeled current business text;
[0084] The model training module 11 is used to determine a first classification label corresponding to the current business text based on a pre-generated target language model and a preset prompt word, train at least one classification model based on the first classification label and the current business text, and determine a second classification label of the current business text based on the classification model;
[0085] The label determination module 12 is used to determine a target classification label of the current business text based on the first classification label and the second classification label of the current business text.
[0086] Based on any optional technical solution in the embodiments of the present invention, optionally, the device further includes:
[0087] A data set division module, used for dividing the current business data set into a test data set and a pre-labeled data set before training at least one classification model based on the first classification label and the current business text;
[0088] A classification label determination module, used to determine a true classification label of a test service text in the test data set based on first labeling information corresponding to the test data set sent by the service terminal;
[0089] A preset prompt word update module is used to update the preset prompt words based on the test business text and the real classification label;
[0090] A classification label updating module, used for updating a first classification label corresponding to a pre-labeled business text in a pre-labeled data set based on an updated preset prompt word and a target language model;
[0091] Model training module 11, including:
[0092] The classification model training submodule is used to train at least one classification model based on the pre-labeled business text and the updated first classification label.
[0093] Based on any optional technical solution in the embodiments of the present invention, optionally, the data set division module includes:
[0094] A business text extraction submodule is used to cluster the current business texts corresponding to the first classification label for each type of first classification label to obtain at least one cluster data subset, and extract at least one current business text from each cluster data subset as a test business text;
[0095] The data set composition submodule is used to compose a test data set from each test business text, and to compose a pre-labeled data set from the remaining business texts in the current business data set excluding the test data set; wherein the number of the remaining business texts is greater than a first preset threshold.
[0096] Based on any optional technical solution in the embodiment of the present invention, optionally, the tag determination module 12 includes:
[0097] A first difficult business text determination submodule is used for, for each pre-labeled business text in the pre-labeled data set, when the first classification label corresponding to the pre-labeled business text does not match the second classification label, to use the pre-labeled business text as the first difficult business text;
[0098] The instruction sending submodule is used to generate a first labeling instruction based on the first difficult business text, and send the first labeling instruction to the business terminal to determine the target classification label corresponding to the first difficult business text based on the first labeling information corresponding to the first difficult business text fed back by the business terminal.
[0099] Based on any optional technical solution in the embodiments of the present invention, optionally, the method further includes:
[0100] A business text update submodule is used to determine whether the number of the first difficult business texts is greater than a second preset threshold before generating a first labeling instruction based on the first difficult business text; if it is greater than the second preset threshold, update the test data set and the pre-labeling data set based on the first difficult business text, and repeat the operations of updating the preset prompt words, updating the first classification label, and training the classification model based on the updated test data set and the pre-labeling data set, update the second classification label based on the trained classification model, and update the first difficult business text based on the updated first classification label and the second classification label;
[0101] Business text update submodule, including:
[0102] An instruction generation unit is used to generate a first marking instruction based on the updated first difficult business text when the updated first difficult business text meets the preset conditions; wherein the preset conditions include that the number of the updated first difficult business text is less than or equal to the second preset threshold and greater than the third preset threshold.
[0103] Based on any optional technical solution in the embodiments of the present invention, optionally, the method further includes:
[0104] A model updating module, configured to update a target language model based on the test business text and the true classification label after determining the true classification label of the test business text in the test data set;
[0105] A second difficult business text determination module is used to determine the third classification label corresponding to the pre-labeled business text in the pre-labeled data set based on the updated target language model; based on the first classification label and the third classification label of the pre-labeled business text, determine the second difficult business text in the pre-labeled business text;
[0106] An instruction generation module is used to generate a second labeling instruction based on the second difficult business text, and send the second labeling instruction to the business terminal to determine the target classification label corresponding to the second difficult business text based on the second labeling information corresponding to the second difficult business text fed back by the business terminal.
[0107] Based on any optional technical solution in the embodiments of the present invention, optionally, the model training module 11 includes:
[0108] The data set generation submodule is used to generate a first training data set and a second training data set based on the current business text and the first classification label; train the first classification model to be trained based on the first training data set to obtain a first classification model, and train the second classification model to be trained based on the second training data set to obtain a second classification model;
[0109] The classification label determination submodule is used to determine the second classification label of the current business text in the second training data set based on the first classification model, and to determine the second classification label of the current business text in the first training data set based on the second classification model.
[0110] The technical solution of the embodiment of the present invention obtains a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text; based on a pre-generated target language model and a preset prompt word, a first classification label corresponding to the current business text is determined, and at least one classification model is trained based on the first classification label and the current business text, thereby determining the first classification label through the target language model and the preset prompt word, thereby eliminating the need for manual labeling, reducing the labor cost and time cost of training the classification model; and determining the second classification label of the current business text based on the classification model; based on the first classification label and the second classification label of the current business text, determining the target classification label of the current business text, so that when determining the target classification label, the first classification label obtained by the target language model and the second classification label obtained by the classification model are combined at the same time, solving the problem of poor labeling accuracy caused by determining the target classification label only through the target language model, and achieving the effect of improving the accuracy of the determined target classification label.
[0111] It is worth noting that in the embodiment of the above-mentioned business text annotation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0112] Figure 5: is a structural schematic diagram of an electronic device that implements the business text annotation method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0113] like Figure 5 As shown, the electronic device 20 includes at least one processor 21, and a memory connected to the at least one processor 21, such as a read-only memory (ROM) 22, a random access memory (RAM) 23, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 21 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 22 or the computer program loaded from the storage unit 28 to the random access memory (RAM) 23. In RAM23, various programs and data required for the operation of the electronic device 20 can also be stored. The processor 21, ROM22 and RAM23 are connected to each other through a bus 24. An input / output (I / O) interface 25 is also connected to the bus 24.
[0114] A number of components in the electronic device 20 are connected to the I / O interface 25, including: an input unit 26, such as a keyboard, a mouse, etc.; an output unit 27, such as various types of displays, speakers, etc.; a storage unit 28, such as a disk, an optical disk, etc.; and a communication unit 29, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 29 allows the electronic device 20 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0115] The processor 21 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 21 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 21 executes the various methods and processes described above, such as a business text annotation method.
[0116] In some embodiments, the business text annotation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 28. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 20 via ROM22 and / or communication unit 29. When the computer program is loaded into RAM23 and executed by the processor 21, one or more steps of the business text annotation method described above can be performed. Alternatively, in other embodiments, the processor 21 can be configured to execute the business text annotation method in any other appropriate manner (for example, by means of firmware).
[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0118] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0119] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0120] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0121] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0122] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0123] This embodiment also provides a computer program product, including a computer program, which, when executed by a processor, implements the business text annotation method provided in any embodiment of the present application.
[0124] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A business text annotation method, characterized in that: include: Acquire a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text; Determine a first classification label corresponding to the current business text based on a pre-generated target language model and a preset prompt word, train at least one classification model based on the first classification label and the current business text, and determine a second classification label for the current business text based on the classification model; A target classification label of the current service text is determined based on the first classification label and the second classification label of the current service text.
2. The method according to claim 1, characterized in that Before training at least one classification model based on the first classification label and the current business text, the method further includes: Dividing the current business data set into a test data set and a pre-labeled data set; Determining a true classification label of a test service text in the test data set based on first labeling information corresponding to the test data set sent by the service terminal; Based on the test business text and the real classification label, updating the preset prompt word; Based on the updated preset prompt word and the target language model, updating the first classification label corresponding to the pre-labeled business text in the pre-labeled data set; The training of obtaining at least one classification model based on the first classification label and the current business text includes: Based on the pre-labeled business text and the updated first classification label, at least one of the classification models is trained.
3. The method according to claim 2, characterized in that The dividing the current business data set into a test data set and a pre-labeled data set includes: For each type of the first classification label, cluster the current business texts corresponding to the first classification label to obtain at least one cluster data subset, and extract at least one current business text from each of the cluster data subsets as a test business text; The test business texts form a test data set, and the remaining business texts in the current business data set excluding the test data set form a pre-marked data set; wherein the number of the remaining business texts is greater than a first preset threshold.
4. The method according to claim 2, characterized in that: The determining, based on the first classification label and the second classification label of the current business text, a target classification label of the current business text includes: For each pre-marked business text in the pre-marked data set, when the first classification label corresponding to the pre-marked business text does not match the second classification label, treating the pre-marked business text as a first difficult business text; A first labeling instruction is generated based on the first difficult business text, and the first labeling instruction is sent to a business terminal to determine a target classification label corresponding to the first difficult business text based on first labeling information corresponding to the first difficult business text fed back by the business terminal.
5. The method according to claim 4, characterized in that Before generating a first marking instruction based on the first difficult business text, the method further includes: Determining whether the number of the first difficult business texts is greater than a second preset threshold; If it is greater than the second preset threshold, the test data set and the pre-labeled data set are updated based on the first difficult business text, and the operations of updating the preset prompt words, updating the first classification label, and training the classification model are repeated based on the updated test data set and pre-labeled data set, the second classification label is updated based on the trained classification model, and the first difficult business text is updated based on the updated first classification label and the second classification label; The generating a first marking instruction based on the first difficult business text includes: When the updated first difficult business text meets the preset conditions, a first marking instruction is generated based on the updated first difficult business text; wherein the preset conditions include that the number of the updated first difficult business text is less than or equal to the second preset threshold and greater than the third preset threshold.
6. The method according to claim 2, characterized in that After determining the true classification labels of the test business texts in the test data set, the method further includes: Based on the test business text and the real classification label, updating the target language model; Determining, based on the updated target language model, a third classification label corresponding to the pre-labeled business text in the pre-labeled data set; Based on the first classification label and the third classification label of the pre-marked business text, determining a second difficult business text in the pre-marked business text; A second labeling instruction is generated based on the second difficult business text, and the second labeling instruction is sent to the business terminal to determine the target classification label corresponding to the second difficult business text based on the second labeling information corresponding to the second difficult business text fed back by the business terminal.
7. The method according to claim 1, characterized in that The training of obtaining at least one classification model based on the first classification label and the current business text, and determining a second classification label of the current business text based on the classification model includes: Generate a first training data set and a second training data set based on the current business text and the first classification label; Training a first classification model to be trained based on the first training data set to obtain a first classification model, and training a second classification model to be trained based on the second training data set to obtain a second classification model; Based on the first classification model, a second classification label of the current business text in the second training data set is determined; based on the second classification model, a second classification label of the current business text in the first training data set is determined.
8. A business text annotation device, characterized in that: include: A data set acquisition module, used to acquire a current business data set to be labeled from a target business system; wherein the current business data set includes at least one unlabeled current business text; A model training module, used to determine a first classification label corresponding to the current business text based on a pre-generated target language model and a preset prompt word, train at least one classification model based on the first classification label and the current business text, and determine a second classification label of the current business text based on the classification model; The label determination module is used to determine a target classification label of the current business text based on the first classification label and the second classification label of the current business text.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the business text annotation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the business text annotation method according to any one of claims 1 to 7 when executed.