Method and apparatus for determining labels for text data

Abstract

The present application proposes a method and device for determining labels of text data. The method includes acquiring text data, extracting text features of the text data, determining at least one label of the text data based on the text features and a multi-label set, wherein the multi-label set includes a regular label set, a long-tail label set, and an unknown label set, and adjusting the multi-label set based on the determined labels of the text data. The solution of the present application can effectively improve the speed, efficiency, and accuracy of multi-label determination of text data in a conversation and reduce costs, accurately extract and update the connection between different labels to reflect the customer's conversation intention.

Description

Background Art

[0002] In the service industry such as catering, the business personnel of the customer service system (customer service system) receive visiting customers in person or through telephone, instant messaging or online platforms in their daily work and solve the problems raised by customers. During the service process, customers will also provide other information about the service to the business personnel. This information can help the company improve its service level.

[0003] After the service conversation between the customer service staff and the customer is over, the customer's conversation intention needs to be extracted and classified so that it can be applied to the downstream system of the customer service business process. In scenarios such as emergency response, by analyzing the conversation data, customer service on-site status information can also be obtained to implement on-site management control.

[0004] Since the extraction and classification of conversation intent needs to meet the requirements of real-time and accuracy, and the information involved in the conversation is very complex, the classification system of conversation data has become increasingly complex, which in turn poses huge challenges to the updating and classification of conversation intent.

[0005] The current conversation intent classification has not yet formed a streamlined, standardized and efficient system. Manually labeling conversation intent using manual methods is inefficient and difficult to control the accuracy. Experience-based preset tags cannot track updated information in conversation intent, nor can they characterize the connections between different tags.

[0006] Therefore, there is a need to improve the existing conversation intent classification schemes. Summary of the invention

[0007] In order to solve at least one of the problems mentioned above, embodiments of the present application provide a method and device for determining a label of text data.

[0008] According to one aspect of the present application, a method for determining labels of text data is proposed, comprising: acquiring text data; extracting text features of the text data; determining at least one label of the text data based on the text features and a multi-label set, the multi-label set comprising a regular label set, a long-tail label set, and an unknown label set; and adjusting the multi-label set based on the determined labels of the text data.

[0009] According to another aspect of the present application, a device for determining a label of text data is proposed, comprising: an acquisition unit, which is configured to acquire text data; a multi-label determination model, including a feature extraction sub-model and a label prediction sub-model, the feature extraction sub-model is configured to extract text features of the text data, the label prediction sub-model is configured to determine at least one label of the text data based on the text features and a multi-label set, the multi-label set including a regular label set, a long-tail label set and an unknown label set; and a monitoring unit, which is configured to adjust the multi-label set based on the determined labels of the text data.

[0010] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by a processor, the method described above is implemented.

[0011] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the method as described above.

[0012] Through the method and device proposed in this application, the speed, efficiency and accuracy of determining the labels of conversation text data for multi-label classification can be effectively improved and the cost can be reduced, the connection between different labels can be accurately extracted and the labels of the determined text data can be updated to reflect the customer's conversation intention. This label determination scheme can support multiple label types and update labels based on changes in time to reflect changes in conversation intentions and topics in the customer's conversation data. In the case of a need to quickly deploy and launch a multi-label determination system, the model building speed and model update and iteration efficiency can be improved, so that the determination results of the multi-label determination system are closer to the customer service scenario content of the service industry, such as the catering industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0014] Figure 1 It is a schematic structural block diagram of a system for determining labels of text data according to an embodiment of the present application.

[0015] Figure 2 A schematic logic block diagram for training a multi-label determination model used in a process for determining labels for text data according to an embodiment of the present application.

[0016] Figure 3 The figure is a schematic flow chart of a method for determining a label of text data according to an embodiment of the present application.

[0017] Figure 4 The figure is a schematic structural block diagram of an electronic device for determining a label of text data according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided to make the scheme of the present application comprehensive and complete and to fully convey the concepts of the exemplary embodiments to those skilled in the art. In the drawings, the size of some elements may be exaggerated or deformed for clarity. The same reference numerals in the drawings represent the same or similar structures, and their detailed descriptions will be omitted.

[0019] In addition, described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application may be put into practice without one or more of the specific details, or other methods, elements, etc. may be adopted. In other cases, known structures, methods or operations are not shown or described in detail to avoid blurring the various aspects of the present application's solution.

[0020] In this article, the customer service scenario of the catering service industry is taken as an example to introduce the method and device for determining the labels of text data for multi-label classification. However, those skilled in the art should understand that the solution proposed in this application is not only applicable to the catering service industry, but can also be applied to other scenarios for extracting the conversation intent of a conversation and determining its labels. These scenarios include but are not limited to instant messaging systems, text data mining systems with multiple information sources, time-related data update and storage systems, etc.

[0021] The conversation intention extracted from the conversation data between the business personnel of the customer service system and the customer, i.e., the conversation topic, usually refers to the aspects and details involved in the feedback information on the catering service provided by the customer in the customer service conversation content. The feedback information may include, for example, the evaluation of the taste of the dishes, the service quality of the store, etc., and may also include the customer's evaluation of the customer service itself.

[0022] Conversational intent can be achieved by marking conversational data with labels. Labels are concise information representing conversational intent or topics, which are equivalent to the attribute dimension or type to which conversational intent or topics belong. Conversational data can have one intent or topic, and then the conversational data can have one label, or the conversational data can be marked with one label. When conversational data has multiple intents or topics, the conversational data can have multiple labels or multiple labels can be used to mark the conversational data. Conversational data with multiple labels indicates that the conversational data has multiple intents or topics, and each topic / intent can be regarded as an aspect or part of the overall topic or overall intent of the conversational data. If the conversational data is regarded as a feature vector, the conversational data with multiple labels indicates that the conversational feature vector has multiple feature components, and each feature component represents a conversational topic or intent that can be marked with a corresponding label. The multi-label marking system is particularly suitable for marking or representing feature vectors (or feature data) with multiple feature components. Therefore, the process of marking multiple conversational intents or topics of conversational data using the multi-label marking system is transformed into the process of determining the labels of the conversational data for multi-label classification. The classification process of marking customer feedback information as simple positive or negative reviews belongs to the binary classification process. Compared with the binary classification process, determining the labels of customer feedback information for multi-label classification belongs to the multi-classification process.

[0023] The multi-label determination and labeling of session data can be carried out by manual labeling or experience-based preset labeling. The speed and efficiency of manual labeling are low, and manual operation is difficult to avoid the control problems of inconsistent standards and accuracy changes. For example, when the type and number of labels exceed the threshold of the number that can be supported by manual operation (for example, hundreds or even thousands of optional labels), manual operation is obviously unrealistic. In addition, manual labeling usually does not support multi-label labeling, which further limits the accuracy of information extracted from the intent or theme of session data. Experience-based preset labels, such as using expert systems to automatically label session data, cannot accurately extract information in customer sessions outside the scope of experience. Manual labeling and experience-based preset labeling mainly focus on isolated label classification, and cannot systematically extract and judge the relationship between each label. For example, whether the labels belong to the same type of labels, whether the labels have different levels and layers, and whether the updated content of session data over time introduces new labels or changes the relationship between labels, these situations cannot be completed by manual labeling and experience-based preset labeling schemes, making them unable to more accurately reflect the real-time situation of customer service scenarios.

[0024] The multi-label determination method and device of the present application are intended to automatically complete the label determination and marking of conversation data after the customer service conversation ends, extract effective information reflecting the conversation topic or intention and key issues at the customer service site from the conversation data in real time or near real time, and save manpower and material costs while improving the accuracy and efficiency of the label determination and marking process.

[0025] The conversation data recorded in the customer service conversation is mainly in the form of text data to facilitate the subsequent text feature extraction and label determination and marking process. For example, customer service conversations provided through short messaging applications such as WeChat and QQ or online methods such as website pages or Weibo, Twitter, etc. and emails can directly generate and record conversation text data. In some cases, for example, the customer service conversation data obtained through a customer service phone call records voice data, and in this case, it is necessary to convert the voice data into text data through speech recognition technology such as ASR technology. The text data of the conversation can be obtained in real time, one by one, or in time and / or batches.

[0026] There are many types of tags involved in text data. For a multi-tag determination system, the tags that can be used for tagging can generally be classified into regular tags, long-tail tags, and unknown tags. Regular tags are tags corresponding to the conversation intent or topic of the text data with the largest sample data volume or the largest proportion in the acquired text data set. Regular tags reflect the most important or most concerned topics and intentions in the text data, and can be pre-set as a regular tag set through experience or data statistics. Long-tail tags are tags corresponding to the intention or topic of text data with a long-tail effect. The text data corresponding to these long-tail tags may not have been valued before, or the number of their sample data is small, or the types of their tags are many, but the total amount of these text data is huge, and their cumulative contribution or total benefit to the conversation intent and topic may exceed those text data corresponding to regular tags. Unknown tags are tags that do not currently belong to any of the existing regular tag sets, and it is temporarily impossible to determine whether they reflect the characteristics of long-tail tags. Neither long-tail tags nor unknown tags belong to the regular tag set, but as the acquired and recorded text data is updated and accumulated, the properties presented by some long-tail tags and unknown tags (such as quantity-related and other feature-related properties) enable them to be promoted to reflect the most important or most concerned conversation topics and intentions (because conversation topics and intentions are also constantly changing) and thus be adjusted to regular tags.

[0027] Usually, at least one conventional tag in the conventional tag set belongs to or constitutes a conventional tag type, or it can be said that these conventional tags have a conventional tag type. In the long-tail tag set composed of long-tail tags and the unknown tag set composed of unknown tags, a long-tail tag type and an unknown tag type can be respectively constituted by multiple long-tail tags and multiple unknown tags, or it can be said that these long-tail tags and unknown tags have a long-tail tag type and an unknown tag type respectively. Therefore, the conventional tag set includes at least one conventional tag type whose number is less than or the same as the number of conventional tags, the long-tail tag set includes at least one long-tail tag type whose number may be less than the number of long-tail tags, and the unknown tag set includes at least one unknown tag type whose number may be less than the number of unknown tags. At the beginning of the operation of the system for multi-tag determination, a preset multi-tag set can be generated based on experience. As the multi-tag determination process proceeds and the system model is updated, various tag sets in the multi-tag set and the tags contained therein can be continuously updated, and more accurate multi-tag determination and prediction of text data can be performed based on the new multi-tag set.

[0028] In addition, the embodiments of the present application also introduce the concept of easily confused tags. Easy to confuse tags do not belong to a specific type of tag, but are tags that are easily labeled or classified incorrectly. Easy to confuse tags constitute an easily confused tag combination. When there are two easily confused tags in a combination, it can be called an easily confused tag pair. Easy to confuse tags may be caused by incorrect label extraction, labeling or setting. Manual labeling is not accurate enough or the parameters of the multi-label determination model are not optimal, which may lead to the appearance of easily confused tags. Easy to confuse tags can be any one or more of regular tags, long-tail tags and unknown tags.

[0029] By distinguishing between regular tags, long-tail tags, unknown tags and easily confused tags, the multi-tag determination method, device and system of the present application can support multiple tags and multi-level tags, and automatically realize the tag determination marking of conversation text data and the update of the multi-tag determination system standard.

[0030] The following is combined with Figure 1 The schematic structural block diagram of the multi-label determination system shown in the figure is used to introduce the process of determining the labels for multi-label classification based on the actual conversation scenario of the catering customer service and the acquired conversation text data.

[0031] The system for performing multi-label determination mainly includes an acquisition unit 110 , a multi-label determination model 120 , a monitoring unit 140 , and an optional filtering unit 130 .

[0032] The multi-label determination process of text data first acquires and records the conversation text data of the to-be-determined label during the customer service conversation by the acquisition unit 110. The text data acquired in real time or in batches at a certain time period constitutes a text data sample set and is input into the multi-label determination model 120. Before being used as input data of the multi-label determination model 120, the text data samples in the text data sample set can be preprocessed so that their format meets the requirements of the multi-label determination model 120.

[0033] The multi-label determination model 120 is the key structure and core functional module of the multi-label determination method and system of the present application, and completes the function of extracting text feature information of text data and predicting the label of the text data based on the extracted text feature information. Among them, the multi-label determination model 120 can include two sub-models, namely the feature extraction sub-model 121 and the label prediction sub-model 122. The feature extraction sub-model 121 completes the function of text feature extraction. Each multi-label determination task can output one or more text data to the multi-label determination model in series or in parallel. The feature extraction sub-model 121 segments the text data and extracts keywords. The text data is usually in the form of a character sequence or a coding sequence, and the extracted keywords can also be in the form of a character sequence or a coding sequence. The label prediction sub-model 122 takes the text feature (vector) output by the feature extraction sub-model 121 as input, and outputs at least one label corresponding to the text data input of the multi-label determination model 120 as at least one label 150 of the determined text data. The labels in the labels 150 of the text data output by the label prediction sub-model 122 meet the multi-label determination criteria preset in the model training process, that is, the labels belong to the preset multi-label set. The multi-label set includes a regular label set, a long-tail label set and an unknown label set. Then, in the determined labels 150 of the text data, if a regular label is marked, the marked regular label should be a regular label of a regular label type in the regular label set; if a long-tail label is marked, the marked long-tail label should be a long-tail label of a long-tail label type in the long-tail label set; and if an unknown label is marked, the marked unknown label should be an unknown label of a unknown label type in the unknown label set.

[0034] The multi-label determination model 120 may adopt a model based on traditional technology or a model based on machine learning or a neural network architecture. The neural network model may, for example, adopt a general DNN or CNN neural network model or a specific type of neural network model. A combination of multiple neural network models may be used to form the multi-label determination model 120 and at least one of the sub-models 121 and 122 therein, or models implemented by different technologies may be used to form the overall multi-label determination model 120 and at least one of the sub-models 121 and 122 therein.

[0035] According to an embodiment of the present application, the BERT network model in the natural language processing NLP model can be used as a feature extraction submodel 121. In order to improve the performance of text feature extraction of text data, a multi-layer neural network can be added after the BERT network model to further extract text features. For example, a BIGRU network model and a BERT network model can be used in series to form a BERT+BIGRU network model to achieve more powerful text feature extraction. The BERT network model and the BIGRU network model need to be trained based on a labeled training data set before use to optimize model parameters. The Attention network can selectively increase the attention for different label types according to the weight, and the Softmax network model can be used as an output layer combined with the Attention network model to form an Attention+Softmax network model as a label prediction submodel 122 to achieve multi-label determination and prediction. In this way, the multi-label determination model 120 uses the BERT+BIGRU+Attention architecture and uses the Softmax network to output the determined label 150 of the text data.

[0036] The monitoring unit 140 is used to process and adjust the multi-label set according to the determined label 150 of the text data. According to the determined label type, the monitoring unit 140 is divided into a long-tail label monitoring subunit 141, an unknown label monitoring subunit 142 and a regular label monitoring subunit 143.

[0037] The long-tail tag monitoring subunit 141 monitors the statistical information of the text data marked as having long-tail tags and adjusts the tag types and / or tags in the long-tail tag set and the regular tag set. For example, the long-tail tag monitoring subunit 141 can add, remove or modify the long-tail tag type or long-tail tag in the long-tail tag set based on the long-tail tag information in the tag 150 of the determined text data and the statistical information of the corresponding text data, and can also examine whether the existing long-tail tags can be promoted or adjusted to regular tags and move these long-tail tags that meet the preset conditions from the long-tail tag set to the regular tag set when the preset conditions are met. The previous long-tail tag set after the update forms a new long-tail tag set 161. After obtaining the long-tail tags removed from the long-tail tag set, the previous regular tag set forms a new regular tag set 163. When the system operation of multi-tag determination is initialized, all long-tail tags in the preset long-tail tag set can be uniformly set to the "other" long-tail tag type and the text data samples marked as having long-tail tags are monitored.

[0038] Similarly, the unknown tag monitoring subunit 142 monitors the statistical information of the text data marked as having unknown tags and adjusts the tag types and / or the tags in the unknown tag set and the conventional tag set. As described above, the unknown tag is not a non-existent tag, but a tag to which the tag type (e.g., conventional tag or long-tail tag) has not been determined. For example, the unknown tag monitoring subunit 142 can add, remove or modify the unknown tag type or unknown tag in the unknown tag set based on the unknown tag information marked in the determined tag 150 of the text data and the statistical information of the corresponding text data, and can also examine whether the existing unknown tags can be promoted and adjusted to conventional tags and move these unknown tags that meet the preset conditions from the unknown tag set to the conventional tag set when the preset conditions are met. The previously updated unknown tag set forms a new unknown tag set 162. After obtaining the unknown tags removed from the unknown tag set, the previous conventional tag set forms a new conventional tag set 163. When the system operation of multi-tag classification is initialized, all unknown tags in the preset unknown tag set can be uniformly set to the "other" unknown tag type and the text data samples marked as having unknown tags are monitored.

[0039] The conventional tag monitoring subunit 143 monitors the statistical information of the text data marked as having conventional tags and adjusts the tag type and / or the tag therein in the conventional tag set. For example, the conventional tag monitoring subunit 143 can increase, remove or modify the conventional tag type or conventional tag in the conventional tag set based on the conventional tag information marked in the tag 150 of the determined text data and the statistical information of the corresponding text data. In general, since the number of historical text data samples corresponding to conventional tags is the largest and the conventional tag types covered are comprehensive enough, the correct rate of the marking result of the multi-label determination model 120 for conventional tags after initial training is relatively high, and the conventional tag type and / or conventional tag in the conventional tag set may not be adjusted. It can be seen from this that the previous conventional tag set forms a new conventional tag set 163 after obtaining the new conventional tags promoted and moved in from the long-tail tag monitoring subunit 141 and the unknown tag monitoring subunit 142, and / or adjusted by the conventional tag monitoring subunit 143.

[0040] The new long-tail tag set 161, the new candidate tag set 162 and the new regular tag set 163 form an updated new multi-tag set 160. As an optimized multi-tag determination standard system, the new multi-tag set 160 can be imported into each tag monitoring sub-unit 141 to 143 of the monitoring unit 140 to iteratively update and adjust the current multi-tag set in the tag monitoring process, and can also verify and re-label the historical text data samples and updated text data samples in the historical training data set to obtain updated historical text data samples and incremental text data samples, so as to form an updated training data set and incremental training data set 170, respectively. Among them, the updated text data samples come from the text data acquired by the acquisition unit 110 at the current moment, and the historical text data samples come from the labeled text data previously acquired by the acquisition unit 110. After multi-tag determination and labeling by the multi-tag determination model 120, the acquired text data becomes the "historical" text data sample at the next moment. These updated training data sets and incremental training data sets 170 may be used to retrain the parameters of the multi-label determination model 120 and its sub-models to obtain updated and more optimal model parameters.

[0041] Before the determined labels 150 of the text data are input into the monitoring unit 140, an optional filtering unit 130 may be used to filter the easily confusing label combinations mentioned above. The filtering unit 130 may realize the calibration of the wrong labels in the determined labels 150 of the text data and the unification and / or further subdivision of the labels of unreasonable markings, obtain more accurate labels corresponding to the text data, and simultaneously obtain more accurate label types and labels in the multi-label set (as well as the long-tail label set, unknown label set and regular label set therein), thereby improving the monitoring effect of the monitoring unit 140. The labels of the text data calibrated by the filtering unit 130 and the corresponding text data form calibrated updated text data samples, which also improve the accuracy of the training data set and thus improve the performance of the trained multi-label determination model 120.

[0042] The monitoring process of the monitoring unit 140 can be performed online in real time or offline in batches to determine whether and how to iteratively update the current multi-label set monitored by the monitoring unit 140 based on the new multi-label set 160 and generate training data for retraining and updating the determination model. The offline method can save system computing resources compared to the online real-time method. Offline monitoring of the multi-label set in synchronization with the update time period of the dishes and / or services of the catering store can also achieve near real-time tracking of the subject and intent of the conversation text data.

[0043] Figure 2 Shows the training process of the multi-label determination model.

[0044] First, in step S210 , historical text data samples are obtained, and labels are manually added to the text data samples to obtain a data set 202 of historical text data samples and their labels. The labels are based on a multi-label determination standard system defined by the initial multi-label set 201 .

[0045] Next, in step S220, these verified historical text data samples with labels are divided into a (historical) training data set and a test data set, and the historical training data set is used to train the multi-label determination model constructed for multi-label determination for multi-label classification scenarios in step S230. In step S240, the test data set generated from a part of the verified historical text data samples in step S220 is used to evaluate whether the performance of the trained multi-label determination model meets the design requirements. When the performance indicators of the determination model meet the design requirements, the training process ends in step S250.

[0046] As time goes by, new text data is continuously input into the multi-label determination model for label determination and prediction. In order for the multi-label determination model to track the changes in conversation text data in customer service scenarios and output more accurate labels, it is necessary to update and further optimize the parameters of the trained multi-label determination model. Since the existing multi-label determination model may output inaccurate labels for newly acquired text data, the update of the multi-label determination model needs to start with calibration of the labels of the determined text data output by the model.

[0047] Step S260 implements Figure 1The filtering function provided by the filtering unit 130 in the filtering step S260. The filtering step S260 can be in real time or every once in a while after one or more multi-label determination tasks for the conversation text data are completed, and in sub-step S261, the business personnel are prompted to manually filter out the text data samples with incorrectly marked labels, calibrate these incorrect labels to correct labels, and record these text data and the corresponding incorrect labels and correct labels in association. In sub-step S262, the combination of correct labels and incorrect labels that are easily confused is further analyzed to find out the frequently occurring label combinations or label pairs. The label of the text data output by the determination model adopts the form of the marked label and the credibility data belonging to the label. The credibility data can be, for example, a probability value. The probability of occurrence of the incorrect label or the probability of occurrence of the text data and its incorrect label recorded in association can be sorted, and the incorrect label with the highest probability of occurrence can be used as the incorrect label of concern. When sorting from large to small by probability, the N incorrect labels (called TopN labels) with the most forward positions can be used with the correct labels to use an association analysis algorithm to find the easily confused label combination. The confusable label information indicates the correctness of the multi-label set configuration and / or the multi-label determination model modeling, and can give a prompt whether the model or label configuration needs to be modified. Then, in step sub-S263, the confusable label combination is converted. The conversion includes merging the confusable label combination into labels of the same type or the same label (i.e., label unification) or converting the confusable label combination into multi-level labels, which are subdivided at different levels. Step sub-S263 can be completed using a predetermined conversion rule and a multi-model integration method.

[0048] The filtering step S260 outputs a data set 203 of text data samples and their labels that have been calibrated with labels. Figure 1 The new multi-label set 160 output by the monitoring unit 140 is used to relabel the labels of the text data samples for the data set 202 of historical text data samples and their labels and the data set 203 of text data samples and their labels after label calibration, respectively, to obtain updated historical text data samples and incremental text data samples and their label data sets, thereby obtaining updated training data sets and test training data sets for retraining the multi-label determination model (step S270).

[0049] The new training dataset generated using the new multi-label set may include updating the historical training dataset based on text data samples that are continuously updated over time to generate an updated training dataset, or generating an incremental training dataset based only on updated text data samples.

[0050] Taking the BERT+BIGRU+Attention network model (Softmax network model as the output layer) introduced above as an example, for text feature extraction, the optimization of its model parameters is less affected by the updated text data samples, so that the historical training data set is sufficient to complete the parameter optimization of the feature extraction sub-model 121. The feature extraction of text data and multi-label prediction in the multi-label determination process are two relatively independent processes. Therefore, the historical training data set composed of labeled historical text data samples can be used to first train the parameters of the feature extraction sub-model 121 composed of the BERT model and the BIGRU model, and freeze the parameters of each network layer of the entire BERT model and the BIGRU model.

[0051] The Attention+Softmax network model pays more attention to the updating of the label marking of the text data over time. Therefore, the historical training data set can be used to perform the initial training of the Attention+Softmax network model to obtain the initial parameters of the model, and then the updated training data set and / or the incremental training data set can be used to perform incremental training on the Attention+Softmax network model to update the parameters of the Attention network model, so that the label prediction sub-model 122 can maintain the conversation topic and intent type of the text data represented by the label type in the preset multi-label set, and identify the new conversation topic and intent type and the label type and label corresponding thereto. If the incremental training data is sufficient to update the training and determine the parameters of the Attention network model, the Attention network model can be incrementally trained using only the incremental training data set. If the incremental training data set is not sufficient to update the parameters of the Attention network model, at least a portion of the historical text data samples can be selected from the historical training data set and the labels marked thereon can be updated to obtain an updated training data set, and the updated training data set and the incremental training data set are used together as the training data set for updating the parameters of the Attention+Softmax network model. It should be noted that the historical text data samples in the historical training data set are data labeled based on the old multi-label set, so if these historical text data samples are not re-labeled with the updated label data in the new multi-label set, the multi-label determination model 120, especially the label prediction sub-model 122, cannot have the ability to recognize new label types and labels. In this way, by incrementally updating the text data samples in the training data and their verified labels, the multi-label determination model 120 can keep the synchronous update and tracking of the topics and intentions in the conversation text data.

[0052] Figure 3 A schematic flow chart of a method for determining a label of text data according to an embodiment of the present application is shown. Figure 1 The system operation process described and based on Figure 2 Similar content to the model training process described will not be repeated.

[0053] The multi-label determination method mainly includes a step S310 of acquiring text data, a step S320 of extracting text features, a step S330 of determining text labels, a step S340 of filtering the determined labels of the text data, a step S350 of monitoring a multi-label set, and a step S360 of updating a multi-label determination model. Some of these steps are optional, such as at least one of steps S340 and S360.

[0054] The method first obtains text data whose labels are to be determined at step S310. The text data comes from conversation text data recorded in a customer service scenario of the catering service industry. The text data can be obtained one by one or in batches.

[0055] Then, in step S320, a Figure 1 The feature extraction submodel 121 of the multi-label determination model 120 shown in extracts text features from the text data obtained in step S310. The text features can be in the form of keywords, for example. The label prediction submodel 122 of the multi-label determination model 120 predicts at least one text label corresponding to the text data in step S330 based on the text features of the extracted text data and a preset or current multi-label set to obtain the label of the text data. As described above, the multi-label set includes a regular label set, a long-tail label set, and an unknown label set. The multi-label determination model 120 used in steps S320 and S330 can be used before being used. Figure 2 The model parameters are determined or updated in the training process shown in (step S200).

[0056] After obtaining the labels of the text data, in step S350, the multi-label set is adjusted and updated by monitoring the relevant attributes of the text data through the determined labels to obtain an updated new multi-label set. The new multi-label set indicates an updated multi-label determination standard system.

[0057] The determined label of each text data has at least one label type, which belongs to one of a regular label, a long-tail label and an unknown label. It can be determined by judging sub-step S354 whether these labels belong to regular labels. If the label does not belong to a regular label (the judgment result of sub-step S354 is "no"), the method determines that the label possessed by the text data is a long-tail label or an unknown label that needs further monitoring. Based on the further classification of the labels, the long-tail labels are monitored and processed in the sub-step S351 of monitoring the long-tail labels, and the unknown labels are monitored and processed in the sub-step S352 of monitoring the unknown labels.

[0058] In sub-step S351, monitor the relevant attributes of the text data with long-tail tags, and adjust and / or re-mark the long-tail tags of the text data online according to the monitoring result, and then adjust the long-tail tag set and the conventional tag set. The monitoring of long-tail tags can first be accumulated and compared for the quantity, frequency and probability-related parameters of the text data with long-tail tags. For example, monitor the quantity of the text data of the long-tail tags belonging to the same long-tail tag type, the frequency of occurrence of these long-tail tags belonging to the same long-tail tag type, and / or the credibility value (such as probability value) of the long-tail tags belonging to the same long-tail tag type possessed by these text data. The long-tail tags with relatively high quantity, frequency or probability parameter values ​​in monitoring can be marked as the long-tail tags of concern online or offline, and as candidate conventional tags, as shown in sub-step S351a. Further monitoring of the long-tail tags of concern can be carried out by the segmented summary, year-on-year and month-on-month mode of quantity or numerical value. When the number of text data samples with long-tail tags of interest accumulates to a certain extent, for example, at least one of the segmented summary quantity, year-on-year quantity and month-on-month quantity of the text data sample exceeds the quantity threshold of the corresponding segmented summary quantity, the quantity threshold of the year-on-year quantity and the quantity threshold of the month-on-month quantity, in sub-step S351b, it is automatically triggered to determine that the long-tail tag of interest can be adjusted / promoted / divided into a regular tag from a candidate regular tag. In the operation of promoting a long-tail tag from a concerned long-tail tag (candidate regular tag) to a regular tag, it can be manually calibrated and marked, or it can be promoted and determined according to a preset judgment rule or algorithm model. The long-tail tag promoted to a regular tag is moved from the long-tail tag set to the regular tag set, and a new long-tail tag set 161 and a new regular tag set 163 are obtained respectively. In general, long-tail tags can be monitored based on all text data samples (full amount samples).

[0059] In sub-step S352, the relevant attributes of the text data with unknown tags are monitored, and the unknown tags of the text data are adjusted and / or re-labeled online according to the monitoring results, so as to adjust the unknown tag set and the conventional tag set. Different from the monitoring of long-tail tags, the monitoring of those unknown tags that need to be concerned can be used as candidate conventional tags. The method of text clustering of the text features of the text data with unknown tags belonging to the same unknown tag type can be used to preliminarily perform label division. Text clustering can be combined with the extraction of core sentences or key sentences in parallel or sequentially to assist in the mining and refinement of unknown tags. Text clustering can form new unknown tags or unknown tag types, and can also refine and calibrate existing unknown tags or unknown tag types. The extraction of core key sentences can optimize the refining results of unknown tags or unknown tag types, and more accurately find the unknown tags of concern. Through text clustering and core key sentence extraction, unknown tags can be marked as unknown tags of concern online or offline, and as candidate conventional tags, as shown in sub-step S352a. Further monitoring of the unknown tags of concern can also be carried out by segmented summary, year-on-year and month-on-month methods of quantity or value. When the number of text data samples with unknown tags of interest accumulates to a certain extent, for example, at least one of the segmented summary quantity, year-on-year quantity and month-on-month quantity of the text data samples exceeds the corresponding segmented summary quantity threshold, year-on-year quantity threshold and month-on-month quantity threshold, sub-step S352b automatically triggers the determination that the unknown tag of interest can be adjusted / promoted / classified as a regular tag from a candidate regular tag. In the operation of promoting an unknown tag from an unknown tag of interest (a candidate regular tag) to a regular tag, it can be manually calibrated and labeled, or it can be promoted and determined according to a preset determination rule or algorithm model. The unknown tag promoted to a regular tag is moved from the unknown tag set to the regular tag set, and a new unknown tag set 162 and a new regular tag set 163 are obtained respectively. In general, monitoring of unknown tags can be performed based on all text data samples (full amount of samples).

[0060] In the determination step S354, if the determination result is "yes", the text data in the multi-label classification result has a conventional label, and the method enters the sub-step S353 for monitoring and processing the conventional label. In the monitoring of conventional labels, those labels that can be used as candidate conventional labels can be directly extracted from the text features of the text data, as shown in sub-step S353a. Further monitoring of candidate conventional labels can also be carried out by segmented summary, year-on-year and month-on-month methods of quantity or value. When the number of text data samples with candidate conventional labels accumulates to a certain extent, for example, at least one of the segmented summary quantity, year-on-year quantity and month-on-month quantity of the text data samples exceeds the corresponding segmented summary quantity threshold, the year-on-year quantity threshold and the month-on-month quantity threshold, in sub-step S353b, it is automatically triggered to determine that the candidate conventional label can be adjusted / promoted / divided into a conventional label. In the operation of promoting candidate conventional labels to conventional labels, it can be manually calibrated and marked, or it can be promoted and determined according to a preset determination rule or algorithm model. The label promoted to a conventional label is directly added to the conventional label set to obtain a new conventional label set 163. Since the number of text data samples predicted to have regular labels is the largest, the accuracy of their multi-label determination results is relatively high, so regular labels can be monitored based on a portion of text data samples (partial samples). Selecting a portion of text data samples to monitor regular labels can avoid the problem that the multi-label determination method and the multi-label determination model do not pay enough attention to long-tail labels and unknown labels, resulting in poor classification performance. Accordingly, using the full amount of sample data to monitor long-tail labels and unknown labels can overcome the problem of too many text data samples corresponding to regular labels and ignoring the attention, extraction and update of other labels other than regular labels.

[0061] According to an embodiment of the present application, conventional tags may not be monitored, and the sub-step S353 of monitoring conventional tags may be omitted. At this time, the update of the conventional tag set is completed by the sub-step S351 of monitoring long-tail tags and the sub-step S352 of monitoring unknown tags. As the text data samples are continuously updated, the conventional tags in the conventional tag set also need to be monitored and updated so that the multi-tag determination method and the determination model can synchronously track the conversation intention and topic of the conversation text data in the customer service scenario of the catering service industry. It can be seen from the above monitoring process that a new conventional tag set can be generated by any one of the three monitoring sub-steps S351, S352 and S353.

[0062] The new long-tail tag set 161, the new unknown tag set 162 and the new regular tag set 163 constitute a new multi-tag set 160, and a reconstructed and optimized multi-tag determination standard system is obtained to serve the multi-tag determination system. The new long-tail tag set 161, the new unknown tag set 162 and the new regular tag set 163 in the new multi-tag set 160 can be provided to three monitoring sub-steps S351, S352 and S353 respectively, so as to iteratively adjust the long-tail tags in the long-tail tag set, the unknown tags in the unknown tag set and the regular tags in the regular tag set based on the multi-tag set that has been adjusted in the previous multi-tag determination process. Through the above process, the monitoring step S350 can enhance the determination and marking capabilities of the multi-tag determination method and system for long-tail tags, unknown tags and regular tags.

[0063] The text data samples in the multi-label determination results after monitoring the corresponding labels form historical text data samples. The text data samples and their adjusted labels that are continuously input into the multi-label determination model for label determination and prediction can be used as updated training data. In step S360, the multi-label determination model is trained again to update the parameters of the multi-label determination model and optimize its performance.

[0064] Step S360 includes sub-steps S361 to S363. Sub-step S361 obtains the text data with excessive labels determined and predicted as described above and the labels calibrated or re-labeled based on the updated multi-label set to form updated training data and constitute an incremental training data subset. The labels of the historical text data samples obtained before the current text data also constitute the updated historical text data samples based on the labels re-labeled or calibrated based on the updated multi-label set, and the corresponding updated training data part is the updated historical training data subset. In sub-step S362, the incremental training data subset and the updated historical training data subset can be combined into an updated training data set, or only the incremental training data subset can constitute the updated training data set (which can also be called an incremental training data set at this time). Then, in sub-step S363, the updated training data set is used to train the multi-label determination model again. The optimized multi-label determination model is used in step S330, which can improve the performance of multi-label determination and prediction of newly input text data samples. For text data monitored and annotated by the system online, it can support downloading and exporting and be used for offline construction or updating of multi-label determination models. Through the release and iteration of the multi-label determination model, the multi-label determination method and system can be made closer to the customer service scenarios of the catering service industry, and the label determination algorithm of text data can be steadily improved for multi-label classification.

[0065] Step S360 and Figure 2There is a similarity with step S270 in . The specific details of updating the multi-label determination model based on the updated entire training data set or only based on the incremental training data set are not described in detail.

[0066] According to an embodiment of the present application, a step S340 of filtering the determined tags of the text data may also be included. For example, step S340 may be located before step S350 of tag monitoring. Figure 2 is similar to the filtering step S260 in . The text data labels output by the multi-label determination model are first judged in the judgment step S341 whether the determined and marked labels are correct. This judgment can be performed after one or more multi-label determination tasks of the conversation text data are completed. Every period of time, the personnel are prompted to manually filter out the text data samples with incorrectly marked labels. When the judgment result is "yes", it indicates that the determined labels are correct, the filtering step S340 is completed, and the step S350 for monitoring the labels is directly entered. When the judgment result is "no", it indicates that at least one of the determined and marked labels is wrong, and the method continues to execute the branches of sub-steps S342 to S344. Here, the wrong label is the confusing label described above. In sub-step S342, the screened wrong labels are calibrated to mark the text data with correct labels, and the wrong labels and correct labels corresponding to these text data are recorded in association. Next, in sub-step S343, the combination of correct labels and wrong labels that are easily confused is analyzed to find out the frequently occurring label combinations or label pairs, and Figure 2 Finally, in sub-step S344 similar to sub-step S263, the easily confused label combination is converted, and the converted labels are used to update the multi-label set for use in the subsequent label monitoring step S350. If there is a filtering step S340, the calibrated or re-labeled labels of the text data samples in step S360 of updating the multi-label determination model will no longer belong to easily confused labels (wrong labels).

[0067] It can be seen that step S340 is similar to step S260, both of which filter the labels of the text data output by the multi-label determination model, calibrate those easily confused labels (i.e., wrong labels), and ensure that the labels of the text data samples are accurate. Figure 2 During the training process of the multi-label determination model, step S260 can be moved to Figure 3Before the step S350 of monitoring the multi-label set in the process, the accuracy of each label in the monitored multi-label set can be guaranteed, and the accuracy of the labels in the updated training data set used to retrain the multi-label determination model can also be guaranteed. Generally speaking, only one of the filtering step S260 in the model training process and the filtering step S340 in the multi-label determination process can be present, and the calibration effect of the filtering step S340 is better.

[0068] Through the method and device proposed in this application, the speed, efficiency and accuracy of determining the labels of conversation text data for multi-label classification can be effectively improved and the cost can be reduced, the connection between different labels can be accurately extracted and the labels of the determined text data can be updated to reflect the customer's conversation intention. This multi-label Penguin East solution can support multiple label types and update labels based on changes in time to reflect changes in conversation intent and topics in the customer's conversation data. In the case of a need to quickly deploy and launch a multi-label determination system, the model building speed and model update and iteration efficiency can be improved, so that the labels determined by the multi-label determination system are closer to the customer service scenario content of the service industry, such as the catering industry.

[0069] Specifically, in the construction and training process of the multi-label determination model, the solution provided by the present application can effectively shorten the training time for multi-level multi-label determination, and while ensuring the recognition accuracy of the original multi-label determination task, it can quickly deploy the recognition of new category labels. For long-tail tags, the solution of the present application builds a complete set of semi-automatic multi-label determination training systems and adds auxiliary monitoring measures, which can better help business personnel manage and maintain long-tail tag sets. For unknown tags, by adopting the method of text clustering + text summarization, business personnel can perceive conversation text data with unknown tags and possible tag classifications with a high probability. For easily confused tag combinations (pairs), a quantitative method is used to analyze and mine the easily confused tag information under the current multi-label type standard system, improve the management efficiency of tags and reduce management costs.

[0070] It should be noted that although several modules or units of the device and system for determining the label of text data are mentioned in the above detailed description, this division is not mandatory. In fact, according to the implementation mode of the present application, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be concretized. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0071] In an exemplary embodiment of the present application, a computer-readable storage medium is also provided, on which a computer program is stored, the program including executable instructions, which can implement the steps of the method for determining the label of text data described in any of the above embodiments when the executable instructions are executed by, for example, a processor. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps of various exemplary embodiments of the present application described in the method for determining the label of text data in this specification.

[0072] The program product for implementing the above method according to the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0073] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0074] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0075] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0076] In an exemplary embodiment of the present application, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the steps of the method for determining a label of text data in any of the above embodiments by executing the executable instructions.

[0077] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0078] Refer to the following Figure 4 The electronic device 400 according to this embodiment of the present application is described. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 4As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410), a display unit 440, etc.

[0080] The storage unit stores a program code, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps of various exemplary embodiments of the present application described in the method for determining the label of text data in this specification. For example, the processing unit 410 can perform the following steps: Figure 2 or Figure 3 Follow the steps shown in .

[0081] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 4201 and / or a cache memory unit 4202 , and may further include a read-only memory unit (ROM) 4203 .

[0082] The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0083] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0084] The electronic device 400 may also communicate with one or more external devices 500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 460. The network adapter 460 may communicate with other modules of the electronic device 400 via the bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0085] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the method for determining the label of text data according to the implementation method of the present application.

[0086] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and embodiments are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the appended claims.

Claims

1. A method for determining a label of text data, comprising: Acquire the text data; Extracting text features of the text data; Determine at least one label of the text data based on the text feature and a multi-label set, the multi-label set including at least one of a regular label set, a long-tail label set, and an unknown label set, the regular label set including at least one regular label, the at least one regular label having at least one regular label type, the long-tail label set including at least one long-tail label, the at least one long-tail label having at least one long-tail label type, the unknown label set including at least one unknown label, the at least one unknown label having at least one unknown label type, and the label of the text data including at least one of the regular label, the long-tail label, and the unknown label; Adjusting the multi-label set based on the determined label of the text data comprises at least one of the following steps: Monitoring text data with long-tail tags to adjust the long-tail tag set and the regular tag set includes: Selecting a candidate regular tag from the long-tail tag set based on at least one of the number of text data having long-tail tags belonging to the same long-tail tag type, the frequency of occurrence of long-tail tags belonging to the same long-tail tag type, and the probability that the text data has long-tail tags belonging to the same long-tail tag type; Monitoring the candidate regular tags to determine long-tail tags to be adjusted to regular tags; monitoring text data with unknown tags to adjust the unknown tag set and the conventional tag set; and Text data having conventional tags is monitored to adjust the set of conventional tags.

2. The method according to claim 1, characterized in that: Monitoring the candidate regular tags to determine the long-tail tags to be adjusted to regular tags further includes: Based on the comparison of at least one of the segmented summary quantity, year-on-year quantity and month-on-month quantity of text data belonging to the same candidate regular tag with the corresponding quantity threshold, the long-tail tag to be adjusted to the regular tag is determined from the candidate regular tags.

3. The method according to claim 1, characterized in that Monitoring text data with unknown labels to adjust the unknown label set further includes: Clustering text features of text data having unknown tags belonging to the same unknown tag type and / or extracting core key sentences to select candidate regular tags from the unknown tag set; The candidate regular labels are monitored to determine unknown labels to be adjusted to regular labels.

4. The method according to claim 3, characterized in that Monitoring the candidate conventional tags to determine unknown tags to be adjusted to conventional tags further includes: Based on the segment summary quantity of text data belonging to the same candidate regular tag, at least one of the year-on-year quantity and the month-on-month quantity is compared with the corresponding quantity threshold, an unknown tag to be adjusted to a regular tag is determined from the candidate regular tags.

5. The method according to claim 1, characterized in that: Monitoring text data with conventional tags to adjust the set of conventional tags further includes: Determining candidate conventional tags based on text features of at least a portion of the text data; The candidate regular tags are monitored to select a regular tag from the candidate regular tags.

6. The method according to claim 1, characterized in that This further includes filtering out confusing tag combinations.

7. The method according to claim 6, characterized in that Filtering easily confusing tag combinations further includes: recording in association the calibration of the labels of the text data, the calibrated incorrect labels of the text data, and the calibrated correct labels of the text data; Based on the associated recorded occurrence probabilities of the erroneous label and the correct label, determining the erroneous label and the correct label as a confusing label combination; The easily confusing tag combinations are merged into the same tag or the easily confusing tag combinations are converted into multi-level tags.

8. The method according to claim 1, characterized in that The label of the text data is determined using a multi-label determination model, wherein the multi-label determination model includes a feature extraction sub-model for extracting text features of the text data and a label prediction sub-model for determining the label of the text data.

9. The method according to claim 8, characterized in that The multi-label determination model is a machine learning model or a neural network model.

10. The method according to claim 9, characterized in that The feature extraction sub-model includes a BERT network model and a BIGRU network model, and the label prediction sub-model includes an Attention network model and a Softmax network model.

11. The method according to claim 8, characterized in that The multi-label determination model is trained using a training dataset before using the multi-label determination model.

12. The method according to claim 11, characterized in that After adjusting the multi-label set based on the determined labels of the text data, updating a training dataset using the text data, the labels of the text data, and the adjusted multi-label set, and retraining the multi-label determination model using the updated training dataset.

13. The method according to claim 1, characterized in that The method further includes adjusting the multiple label sets adjusted in a previous label determination process based on the determined labels of the text data.

14. The method according to claim 1, characterized in that The text data is conversation text in a customer service scenario in the catering industry.

15. A device for performing multi-label classification on text data, comprising: an acquiring unit, configured to acquire the text data; A multi-label determination model, comprising a feature extraction submodel and a label prediction submodel, wherein the feature extraction submodel is configured to extract text features of the text data, and the label prediction submodel is configured to determine at least one label of the text data based on the text features and a multi-label set, wherein the multi-label set comprises at least one of a regular label set, a long-tail label set, and an unknown label set, wherein the regular label set comprises at least one regular label, wherein the at least one regular label has at least one regular label type, the long-tail label set comprises at least one long-tail label, wherein the at least one long-tail label has at least one long-tail label type, the unknown label set comprises at least one unknown label, wherein the at least one unknown label has at least one unknown label type, and the label of the text data comprises at least one of the regular label, the long-tail label, and the unknown label; as well as A monitoring unit is configured to adjust the multi-label set based on the determined label of the text data, the monitoring unit comprising a long-tail label monitoring subunit, an unknown label monitoring subunit and a regular label monitoring subunit, wherein: The long-tail tag monitoring subunit is configured to monitor text data with long-tail tags to adjust the long-tail tag set and the regular tag set, including: selecting candidate regular tags from the long-tail tag set based on at least one of the number of text data with long-tail tags belonging to the same long-tail tag type, the occurrence frequency of long-tail tags belonging to the same long-tail tag type, and the probability that text data has long-tail tags belonging to the same long-tail tag type; monitoring the candidate regular tags to determine the long-tail tags to be adjusted to regular tags; The unknown tag monitoring subunit is configured to monitor text data with unknown tags to adjust the unknown tag set and the regular tag set; The regular tag monitoring subunit is configured to monitor text data with regular tags to adjust the regular tag set.

16. The device according to claim 15, characterized in that The device further comprises a filtering unit configured to filter confusing tag combinations.

17. The device according to claim 15, characterized in that The multi-label determination model is a machine learning model or a neural network model.

18. The device according to claim 17, characterized in that The feature extraction sub-model includes a BERT network model and a BIGRU network model, and the label prediction sub-model includes an Attention network model and a Softmax network model.

19. The device according to claim 15, characterized in that The multi-label determination model is trained using a training dataset before using the multi-label determination model.

20. The device according to claim 19, characterized in that After adjusting the multi-label set based on the determined labels of the text data, updating a training dataset using the text data, the labels of the text data, and the adjusted multi-label set, and retraining the multi-label determination model using the updated training dataset.

21. The device according to claim 15, characterized in that The monitoring unit is further configured to adjust the multi-label set adjusted in a previous multi-label determination process based on the determined labels of the text data.

22. A computer-readable storage medium having a computer program stored thereon, the computer program comprising executable instructions, and when the executable instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

23. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the executable instructions to implement the method according to any one of claims 1 to 14.

Images