Text data cleaning method, device, storage medium and equipment

Abstract

The present application discloses a text data cleaning method, device, storage medium and equipment. The method comprises: first, after training a text classification model with text training data of a clean category, inputting target text data and text training data of a clean category into the model for a first data cleaning, and then judging whether the first cleaning result meets a preset condition for stopping data cleaning; if so, using it as the cleaning result of the target text data; if not, using the text training data of the clean category and the target text data remaining after cleaning to train the text classification model for a second time, and using the updated text classification model and text training data of the clean category to clean the target text data for a second time, and so on, until the Mth cleaning result obtained meets the preset condition for stopping data cleaning, and using the Mth cleaning result as the cleaning result of the target text data, thereby improving the data cleaning effect.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text data cleaning method, device, storage medium and equipment. Background Art

[0002] With the continuous breakthroughs in artificial intelligence technology and the increasing popularity of various smart terminal devices, human-computer interaction appears more and more frequently in people's daily work and life, which can bring great convenience to people. However, in human-computer interaction, data inconsistency is more difficult to handle and will have a greater negative impact on semantic understanding. Therefore, effective cleaning of text data is needed to improve the consistency of text data.

[0003] At present, there are usually two methods for cleaning text data in human-computer interaction: one is to clean based on keywords, but this cleaning method requires re-sorting data and summarizing keywords for each classification category of each language, which is time-consuming and labor-intensive. In addition, some languages ​​cannot list keywords in all aspects, which will lead to insufficient data cleaning. In addition, there will be overcorrection, resulting in a waste of text data resources. Another commonly used text data cleaning method is to use model thresholds for cleaning. Although the time and labor costs are relatively low, the model thresholds need to be manually determined. For different classification categories of different languages, the model thresholds may be different, which may cause the time and labor costs to increase exponentially, and the cleaning efficiency is still low. In addition, this cleaning method requires additional training to introduce the cleaning model, and the training of the cleaning model requires the introduction of text category confusion data, which may have a negative impact on the results and lead to poor cleaning effects. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a text data cleaning method, device, storage medium and equipment, which can achieve accurate and rapid cleaning of text data, thereby improving the effect of text data cleaning.

[0005] The present application embodiment provides a text data cleaning method, including:

[0006] Obtain target text data to be cleaned; the target text data is mixed with some clean category text data;

[0007] After training the text classification model using the text training data of the clean category, inputting the target text data and the text training data of the clean category into the text classification model for first data cleaning to obtain a first cleaning result; the first cleaning result includes the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning;

[0008] Determine whether the first cleaning result meets the preset condition for stopping cleaning data; if so, use the first cleaning result as the final cleaning result corresponding to the target text data; if not, use the text training data of the clean category and the target text data remaining after the cleaning to train the text classification model for the second time to obtain an updated text classification model, and use the updated text classification model and the text training data of the clean category to perform a second data cleaning on the target text data to obtain a second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition for stopping cleaning data, then use the Mth cleaning result as the final cleaning result corresponding to the target text data, where M is a positive integer greater than 0.

[0009] In a possible implementation, the method further includes:

[0010] Obtain text training data of N clean categories; N is a positive integer greater than 0; the category to which the text training data of the clean category obfuscated in the target text data belongs is not greater than N;

[0011] The initial text classification model is trained using the text training data of the N clean categories and the cross entropy loss function to obtain the text classification model.

[0012] In a possible implementation, the method further includes:

[0013] Get text verification data belonging to N clean categories;

[0014] Inputting the text verification data into the text classification model to obtain a clean category prediction result corresponding to the text verification data;

[0015] When the clean category prediction result corresponding to the text verification data meets the preset verification condition, the updating of the text classification model is stopped to obtain the verified text classification model.

[0016] In a possible implementation, inputting the target text data and the clean category text training data into the text classification model for first data cleaning to obtain a first cleaning result includes:

[0017] Inputting the text training data of the clean category into the text classification model, obtaining a first statistic of the text training data of the clean category, and calculating a control limit value of the first statistic;

[0018] Inputting the target text data into the text classification model to obtain a second statistic for each text data in the target text data under the clean category;

[0019] The second statistic is compared with the control limit value, and based on the comparison result, it is determined whether a text data in the target text data corresponding to the second statistic belongs to the clean category. If so, it is cleaned from the target text data to obtain a first cleaning result.

[0020] In a possible implementation, the second statistic is compared with the control limit value, and according to the comparison result, it is determined whether a text data in the target text data corresponding to the second statistic belongs to a clean category, and if so, it is cleaned from the target text data to obtain a first cleaning result, including:

[0021] It is determined whether the absolute value of the second statistic is less than the absolute value of the control limit value. If so, it means that a text data in the target text data corresponding to the second statistic belongs to the clean category. After cleaning it from the target text data, the first cleaning result is obtained.

[0022] In a possible implementation, the second statistic is compared with the control limit value, and according to the comparison result, it is determined whether a text data in the target text data corresponding to the second statistic belongs to a clean category, and if so, it is cleaned from the target text data to obtain a first cleaning result, including:

[0023] Calculate the absolute value of the difference between the second statistic corresponding to each text data in the target text data and the control limit value, and determine the target second statistic corresponding to the absolute value less than a preset absolute value threshold;

[0024] After determining that the text data in the target text data corresponding to the target second statistic belongs to the clean category and cleaning it from the target text data, a first cleaning result is obtained.

[0025] In one possible implementation, the preset condition for stopping data cleaning is that after two consecutive data cleanings, the total amount of text data belonging to the clean category cleaned from the target text data fluctuates by less than a preset number of items; or the preset condition for stopping data cleaning is that the amount of text data remaining after the target text data is cleaned accounts for no less than a preset proportion of the total amount of original text data of the target text data.

[0026] The present application also provides a text data cleaning device, including:

[0027] A first acquisition unit is used to acquire target text data to be cleaned; the target text data is mixed with some clean category text data;

[0028] A first cleaning unit is used for inputting the target text data and the clean class text training data into the text classification model for first data cleaning after training the text classification model with the clean class text training data to obtain a first cleaning result; the first cleaning result includes the clean class text data cleaned from the target text data and the target text data remaining after cleaning;

[0029] The second cleaning unit is used to determine whether the first cleaning result meets the preset condition of stopping cleaning data; if so, the first cleaning result is used as the final cleaning result corresponding to the target text data; if not, the text classification model is trained for a second time using the text training data of the clean category and the target text data remaining after the cleaning to obtain an updated text classification model, and the target text data is cleaned for a second time using the updated text classification model and the text training data of the clean category to obtain a second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning data, then the Mth cleaning result is used as the final cleaning result corresponding to the target text data, where M is a positive integer greater than 0.

[0030] In a possible implementation manner, the device further includes:

[0031] A second acquisition unit is used to acquire text training data of N clean categories; N is a positive integer greater than 0; the category to which the text training data of the clean category obfuscated in the target text data belongs is not greater than N;

[0032] The training unit is used to train the initial text classification model using the text training data of the N clean categories and the cross entropy loss function to obtain the text classification model.

[0033] In a possible implementation manner, the device further includes:

[0034] A third acquisition unit is used to acquire text verification data belonging to N clean categories;

[0035] An obtaining unit, used for inputting the text verification data into the text classification model to obtain a clean category prediction result corresponding to the text verification data;

[0036] The updating unit is used to stop updating the text classification model when the clean category prediction result corresponding to the text verification data meets the preset verification condition, so as to obtain the verified text classification model.

[0037] In a possible implementation, the first cleaning unit includes:

[0038] A first input subunit, used for inputting the text training data of the clean category into the text classification model, obtaining a first statistic of the text training data of the clean category, and calculating a control limit value of the first statistic;

[0039] A second input subunit, used for inputting the target text data into the text classification model to obtain a second statistic of each text data in the target text data under the clean category;

[0040] The cleaning subunit is used to compare the second statistic with the control limit value, and determine whether a text data in the target text data corresponding to the second statistic belongs to the clean category based on the comparison result. If so, clean it from the target text data to obtain the first cleaning result.

[0041] In a possible implementation, the cleaning subunit is specifically used for:

[0042] It is determined whether the absolute value of the second statistic is less than the absolute value of the control limit value. If so, it means that a text data in the target text data corresponding to the second statistic belongs to the clean category. After cleaning it from the target text data, the first cleaning result is obtained.

[0043] In a possible implementation, the cleaning subunit includes:

[0044] A calculation subunit, used to calculate the absolute value of the difference between the second statistic corresponding to each text data in the target text data and the control limit value, and determine the target second statistic corresponding to the absolute value less than a preset absolute value threshold;

[0045] The determination subunit is used to determine whether the text data in the target text data corresponding to the target second statistic belongs to a clean category, and obtain a first cleaning result after cleaning it from the target text data.

[0046] In one possible implementation, the preset condition for stopping data cleaning is that after two consecutive data cleanings, the total amount of text data belonging to the clean category cleaned from the target text data fluctuates by less than a preset number of items; or the preset condition for stopping data cleaning is that the amount of text data remaining after the target text data is cleaned accounts for no less than a preset proportion of the total amount of original text data of the target text data.

[0047] The embodiment of the present application also provides a text data cleaning device, including: a processor, a memory, and a system bus;

[0048] The processor and the memory are connected via the system bus;

[0049] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned text data cleaning method.

[0050] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned text data cleaning method.

[0051] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned text data cleaning method.

[0052] The embodiments of the present application provide a text data cleaning method, device, storage medium and equipment, firstly obtaining target text data to be cleaned; wherein, part of the text data of the clean category is mixed in the target text data, and then after training the text classification model with the text training data of the clean category, the target text data and the text training data of the clean category are input into the text classification model for the first data cleaning to obtain the first cleaning result; wherein, the first cleaning result includes the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning; then, judging whether the first cleaning result meets the preset condition of stopping cleaning the data; if so, taking the first cleaning result as the final cleaning result corresponding to the target text data; if not, training the text classification model for the second time with the text training data of the clean category and the target text data remaining after cleaning to obtain the updated text classification model, and performing the second data cleaning on the target text data with the updated text classification model and the text training data of the clean category to obtain the second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning the data, then taking the Mth cleaning result as the final cleaning result corresponding to the target text data, wherein M is a positive integer greater than 0.

[0053] It can be seen that since the present application can complete the cleaning of the target text data by only using the text training data of the clean category for text classification model training, it not only has strong logical rationality and does not require excessive human intervention, but also can achieve accurate and rapid cleaning of the target text data, thereby improving the effect of text data cleaning, which is facilitating subsequent text-related tasks such as text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A diagram of the training process of a text classification model provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of a text data cleaning method provided in an embodiment of the present application;

[0057] Figure 3 The overall process diagram of text data cleaning provided by the embodiment of the present application;

[0058] Figure 4 A schematic diagram of the composition of a text data cleaning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In the process of human-computer interaction, data consistency is more difficult to handle and will have a greater negative impact on semantic understanding. This is because the consistency of the text data used in the training process will greatly affect the effect of the subsequent text classification task, that is, the semantic understanding ability of the model. For example, if the training data of the intent classification model has categories A and B, and some data in category A is incorrectly marked as category B, the understanding ability of the intent classification model trained based on this data for category A will be greatly negatively affected in the actual application reasoning process.

[0060] For example: In the current commonly used human-computer interaction, semantic understanding is mostly for designated skill category understanding, such as music, film, navigation, smart home, etc., and conversations outside of these categories of skills often do not want to be mistakenly absorbed by designated category skills (incorrectly judged as the current category of skills). Therefore, when training a certain category of skills, a "reject" category is added to indicate that the text does not belong to the current category of skills. In order to be suitable for practical applications, the data in the "reject" category is a large amount of actual data collected, covering a wide range. Therefore, the rejected data usually has some confusion with the in-set data, where the in-set data refers to the data belonging to a certain category of skills. Training the intent classification model of a certain category of skills based on the current data will have a significant negative impact on the intent classification model of the category of skills.

[0061] For example, in the current training process of text classification models, Figure 1As shown in the figure, the text data that meets the training requirements after regularization is passed through the pre-training model (BERT, ALBERT), and the classification probability is calculated by softmax, and then the multi-classification cross entropy loss is calculated using the training data labels, and the weights are updated by back propagation. In this way, the fine-tuning of a skill intent classification model is completed. The current approach is only applicable to the case where all categories of data for a skill do not have text category confusion. For example, music skills have categories such as "PLAY", "CHANGE", "RANDOM", and "OTHERS", and all categories of data do not have mislabeling. However, in actual applications, except for a small amount of artificially constructed or manually quality-checked data, most data are mislabeled or mislabeled, such as the out-of-set (OTHERS category) data collected in human-computer interaction, which contains some "PLAY", "CHANGE", and "RANDOM" data. Using uncleaned data to train the intent classification model is likely to have a negative effect on the model effect.

[0062] Therefore, effective cleaning of text data is required to improve the consistency of text data, so as to reduce the negative impact on the intent classification model from the source and improve the semantic understanding ability under the current business logic.

[0063] At present, the methods for cleaning text data in human-computer interaction usually include the following two methods:

[0064] (1) Text data cleaning based on keywords.

[0065] Specifically, we observe the representative keywords in each category of skill data, such as "play" and "music" in music, "navigation" and "go" in maps, and use the search results of keywords in the rejected data to eliminate the corresponding category skill-related data to complete the cleaning of the rejected data.

[0066] However, this cleaning method requires re-sorting data and summarizing keywords for each classification category in each language, which is time-consuming and labor-intensive. In addition, the keywords for a certain category of skills in some languages ​​cannot be fully listed, resulting in insufficient data cleaning. For example, the words for "get to", "reach", "arrive", "navigate to", "navigation" and so on in the English navigation category skills represent "to", "go", and "navigation". In addition, using keywords to clean data is likely to be an overreaction. Overly strict keyword cleaning rules will largely clean up data that is not in the current skill set. To some extent, it is also a waste of text data resources.

[0067] (2) Use model threshold to clean text data.

[0068] Specifically, the "base model (such as BERT, ALBERT and other pre-trained models) + binary classification" method can be used to complete the training of the binary classification model using text data with strong consistency and data with text category confusion, which can be defined as a cleaning model. Then use the cleaning model to infer data outside the set, and manually set a threshold for the classification result score of the data to clean the data outside the set. Use the obtained data to train the intent classification model of the current skill and verify the effect. If the effect is ideal, end the cleaning. If the effect is not ideal, redefine the threshold and train the intent model to verify the effect until the ideal effect is achieved on the verification set.

[0069] Although this cleaning method is relatively less time-consuming and labor-intensive than the first cleaning solution, the "artificial threshold setting" still requires a lot of attempts by researchers who are familiar with related work to ensure the effectiveness of this method. And for different skills in different languages, the thresholds may be different, so the time and labor costs may increase exponentially, and the cleaning efficiency is still low. In addition, this cleaning method requires additional training to introduce a cleaning model, and the training of this cleaning model requires the introduction of data with text category confusion, which may have a negative impact on the results and lead to poor cleaning effects.

[0070] It can be seen that the above two existing text data cleaning methods not only have poor versatility and poor cleaning effects, but also have high manpower and time costs. Therefore, how to complete the cleaning of data with text category confusion with high quality without requiring too much manpower and time investment, so as to improve the quality of training data and reduce the negative impact of data on semantic understanding models is an urgent problem to be solved.

[0071] To solve the above defects, the present application provides a text data cleaning method, firstly, obtaining target text data to be cleaned; wherein, part of the text data of the clean category is mixed in the target text data, and then after training a text classification model with the text training data of the clean category, the target text data and the text training data of the clean category are input into the text classification model for the first data cleaning to obtain a first cleaning result; wherein, the first cleaning result includes the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning; then, judging whether the first cleaning result meets the preset condition of stopping cleaning the data; if so, taking the first cleaning result as the final cleaning result corresponding to the target text data; if not, training the text classification model for the second time with the text training data of the clean category and the target text data remaining after cleaning to obtain an updated text classification model, and performing a second data cleaning on the target text data with the updated text classification model and the text training data of the clean category to obtain a second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning the data, then taking the Mth cleaning result as the final cleaning result corresponding to the target text data, wherein M is a positive integer greater than 0.

[0072] It can be seen that since the present application can complete the cleaning of the target text data by only using the text training data of the clean category for text classification model training, it not only has strong logical rationality and does not require excessive human intervention, but also can achieve accurate and rapid cleaning of the target text data, thereby improving the effect of text data cleaning, which is facilitating subsequent text-related tasks such as text classification.

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] First embodiment

[0075] See also Figure 2 , is a flow chart of a text data cleaning method provided in this embodiment, the method comprising the following steps:

[0076] S201: Obtain target text data to be cleaned; wherein the target text data is mixed with some clean category text data.

[0077] In this embodiment, any text data that uses this embodiment to clean obfuscated data is defined as the target text data to be cleaned, that is, the target text data is obfuscated with some clean category text data, and it is necessary to use the cleaning method provided by this application to clean all of this part of the data from the target text data. It should be noted that this embodiment does not limit the language type of the target text data, for example, the target text data can be Chinese text data, English text data, or Japanese text data, etc.; this embodiment also does not limit the source and acquisition method of the target text data, for example, the target text data can be the result from speech recognition, or it can be news text data obtained from a network platform, etc.

[0078] It is understandable that there is no limitation on the method of obtaining target text data. Figure 3 As shown, a large amount of existing text data of various categories (such as "PLAY", "CHANGE", "RANDOM", "OTHERS" and other categories in music skills) can be first obtained as training text data. Then, these training text data are preliminarily screened to distinguish the clean category text data and the target text data to be cleaned. Among them, the clean category text data refers to the text data with strong consistency that is artificially constructed or manually inspected and has almost no mislabeling. There are usually n categories of this data (n is a positive integer greater than 0), such as the three categories of "PLAY", "CHANGE", and "RANDOM" in music skills, and n is 3 at this time. The target text data to be cleaned refers to the collected data that may have label confusion, that is, mislabeling. This data is a single category (such as the "OTHERS" category in music skills) text data, but there may be m types of mislabeled category data (0≤m≤n).

[0079] S202: After training the text classification model using the text training data of the clean category, the target text data and the text training data of the clean category are input into the text classification model for a first data cleaning to obtain a first cleaning result; wherein the first cleaning result includes the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning.

[0080] In this embodiment, after a large amount of text data of clean categories is screened out from the training text data, N text data of clean categories can be selected as text training data of N clean categories, where N is a positive integer greater than 0, and the category to which the text training data of the clean categories obfuscated in the target text data belongs is not greater than N. Then, the initial text classification model is trained using the text training data of the N clean categories and the cross entropy loss function to obtain a text classification model.

[0081] Specifically, the text training data of N clean categories can be input into the initial text classification model (the specific model structure is not limited, and models such as BERT and ALBERT can be used as the initial text classification model) to obtain the corresponding features. After softmax classification, the cross entropy loss function can be used under the guidance of the corresponding category label to complete the weight update, that is, the model training, and obtain the text classification model.

[0082] Next, after a large amount of clean category text data is screened out from the training text data, N clean category text data are first selected as N clean category text verification data, and then, after the text verification data is input into the text classification model, the clean category prediction result corresponding to the text verification data is obtained. When the clean category prediction result corresponding to the text verification data meets the preset verification condition (the specific content is not limited, such as the preset verification condition can be set to the prediction accuracy reaching a preset percentage (such as 95%), etc.), the text classification model is stopped from being updated to obtain a verified text classification model.

[0083] On this basis, after obtaining the target text data to be cleaned through step S101, the target text data and the text training data of the clean category can be input into the text classification model for the first data cleaning to obtain the first cleaning result for executing the subsequent step S103; wherein, the first cleaning result includes but is not limited to the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning.

[0084] Specifically, we can first input the text training data of the clean category into the text classification model to obtain the first statistic of the total amount of data in the text training data of the clean category (for example, if there are 1000 training data in the PLAY category of music skills, then the total amount of data is 1000 data), and define it as The specific form is a numerical value. And the first statistic is calculated using existing or future calculation methods. The control limit values ​​of, for example, can be calculated using the KDE (Kernel density estimation) method The control limit values ​​are defined as

[0085] Then, the target text data is also input into the text classification model to obtain the second statistic of each text data in the target text data under the clean category, and it is defined as The specific form of expression is a numerical value. For example, the OTHERS category of music skills can get a data statistic on the PLAY category benchmark. If the amount of target text data is p (a positive integer greater than 0), it needs to be calculated p times to get all the statistics of the target text data under the corresponding clean category as the benchmark.

[0086] The above mentioned and The specific calculation method is as follows:

[0087]

[0088]

[0089]

[0090] Among them, h global (X category (Train)) represents the features of a clean category training data in the skill after passing through the model; X category (Train) represents a clean category training data; h global (Y other (Train)) represents the features of a text data in the target text data after passing through the model; Y other (Train) represents a piece of text data in the target text data. global (X category (i)) represents the features of the i-th data in the corresponding text category after being passed through the model, such as the i-th data in the PLAY category of music skills. j represents the total amount of data in the current text category.

[0091] Next, the second statistic can be and control limit values A comparison is performed, and based on the comparison result, it is determined whether a piece of text data in the target text data corresponding to the second statistic belongs to the clean category. If so, it can be cleaned from the target text data to obtain the first cleaning result.

[0092] Specifically, an optional implementation method is to determine the second statistic Is the absolute value of less than the control limit value? The absolute value of Is it true? If so, it means the second statistic If a piece of text data in the corresponding target text data belongs to the clean category, it needs to be cleaned out of the target text data; otherwise, If true, then the second statistic A text data in the corresponding target text data does not belong to the clean category and does not need to be cleaned from the target text data. It can be understood that this process requires n×p comparisons to complete the first cleaning of the target text data and obtain the first cleaning result. Where n is the number of clean data types and p is the total amount of target text data.

[0093] Alternatively, another optional implementation method is to first calculate the absolute value of the difference between the second statistic corresponding to each text data in the target text data and the control limit value, and determine the target second statistic corresponding to the absolute value that is less than a preset absolute value threshold (the specific value is not limited and can be set according to actual conditions and experience, for example, it can be set to 0.2); then it can be determined that the text data in the target text data corresponding to the target second statistic belongs to the clean category, and after cleaning it from the target text data, the first cleaning result is obtained.

[0094] S203: Determine whether the first cleaning result meets the preset condition for stopping cleaning the data; if so, use the first cleaning result as the final cleaning result corresponding to the target text data; if not, use the text training data of the clean category and the target text data remaining after cleaning to train the text classification model for the second time to obtain an updated text classification model, and use the updated text classification model and the text training data of the clean category to perform a second data cleaning on the target text data to obtain a second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition for stopping cleaning the data, then use the Mth cleaning result as the final cleaning result corresponding to the target text data, where M is a positive integer greater than 0.

[0095] In this embodiment, after obtaining the first cleaning result through step S102, it can be further determined whether the first cleaning result meets the preset condition for stopping cleaning the data, wherein the specific content of the preset condition for stopping cleaning the data is not limited, and can be set according to actual conditions and experience values. For example, the preset condition for stopping cleaning the data can be set to that after two consecutive data cleanings, the total amount of text data belonging to the clean category cleaned from the target text data fluctuates less than a preset number (the specific value is not limited, such as 100 items); or the preset condition for stopping cleaning the data can be set to that the amount of text data remaining after the target text data is cleaned accounts for not less than a preset proportion in the total amount of original text data of the target text data (the specific value is not limited, such as the preset proportion can be 85%, etc.).

[0096] like Figure 3As shown, if yes, that is, the first cleaning result meets the preset condition for stopping cleaning the data, then the first cleaning result can be used as the final cleaning result corresponding to the target text data, and a text classification model after data cleaning can be obtained.

[0097] On the contrary, if not, that is, the first cleaning result does not meet the preset conditions for stopping cleaning the data, then the text classification model can be trained for a second time using the text training data of the clean category and the target text data remaining after cleaning to obtain an updated text classification model, and the target text data can be cleaned for a second time using the updated text classification model and the text training data of the clean category to obtain a second cleaning result; and so on, until the Mth cleaning result obtained meets the preset conditions for stopping cleaning the data, then the Mth cleaning result can be used as the final cleaning result corresponding to the target text data. In this way, through the cyclic cleaning method, the target text data can be cyclically cleaned again and again. When the preset conditions for stopping cleaning the data are met, it means that the current data has reached the maximum cleaning degree, and the final data cleaning result and the cleaned text classification model can be obtained, such as Figure 3 shown.

[0098] In this way, when the target text data is cleaned by executing the above steps S201-S203, there is no need to introduce additional models. Only the text data of clean categories with strong consistency can be used to complete the efficient and high-quality cleaning of data with text category confusion. There is no need for excessive human intervention, which can greatly reduce the investment in manpower and time costs.

[0099] To facilitate understanding of the text data cleaning method provided by the present application, the telephone skill data is now taken as an example of existing training text data for explanation. First, the existing training text data can be preliminarily screened. Three categories are obtained as categories with strong data consistency, including "making a phone call", "video call", and "replay", collectively referred to as in-set data, and one category has a large amount of label confusion data, referred to as out-of-set data (others). Then, the in-set category data is used to complete the training and verification of the text classification model, and the model is used to perform the first data cleaning on the out-of-set data (others) (i.e., the statistics of the others data on the in-set category are calculated separately, and compared with the control limits of each category to complete the first round of data cleaning), and then it is determined whether to perform multiple rounds of cleaning by judging whether the cleaning result meets the preset conditions for stopping data cleaning, until the cleaning result that meets the preset conditions for data cleaning and the cleaned text classification model are obtained.

[0100] In summary, the present embodiment provides a text data cleaning method, firstly obtaining target text data to be cleaned; wherein, part of the text data of the clean category is confused in the target text data, and then after the text classification model is trained with the text training data of the clean category, the target text data and the text training data of the clean category are input into the text classification model for the first data cleaning to obtain the first cleaning result; wherein, the first cleaning result includes the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning; then, judging whether the first cleaning result meets the preset condition of stopping cleaning data; if so, taking the first cleaning result as the final cleaning result corresponding to the target text data; if not, training the text classification model for the second time with the text training data of the clean category and the target text data remaining after cleaning to obtain an updated text classification model, and performing the second data cleaning on the target text data with the updated text classification model and the text training data of the clean category to obtain the second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning data, then taking the Mth cleaning result as the final cleaning result corresponding to the target text data, wherein M is a positive integer greater than 0.

[0101] It can be seen that since the present application can complete the cleaning of the target text data by only using the text training data of the clean category for text classification model training, it not only has strong logical rationality and does not require excessive human intervention, but also can achieve accurate and rapid cleaning of the target text data, thereby improving the effect of text data cleaning, which is facilitating subsequent text-related tasks such as text classification.

[0102] Second embodiment

[0103] This embodiment will introduce a text data cleaning device. For related content, please refer to the above method embodiment.

[0104] See also Figure 4 , is a schematic diagram of the composition of a text data cleaning device provided in this embodiment, the device 400 includes:

[0105] The first acquisition unit 401 is used to acquire target text data to be cleaned; the target text data is mixed with some clean category text data;

[0106] A first cleaning unit 402 is used for inputting the target text data and the clean class text training data into the text classification model for first data cleaning after training the text classification model with the clean class text training data to obtain a first cleaning result; the first cleaning result includes the clean class text data cleaned from the target text data and the target text data remaining after cleaning;

[0107] The second cleaning unit 403 is used to determine whether the first cleaning result meets the preset condition of stopping cleaning data; if so, the first cleaning result is used as the final cleaning result corresponding to the target text data; if not, the text classification model is trained for a second time using the text training data of the clean category and the target text data remaining after the cleaning to obtain an updated text classification model, and the target text data is cleaned for a second time using the updated text classification model and the text training data of the clean category to obtain a second cleaning result; and so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning data, then the Mth cleaning result is used as the final cleaning result corresponding to the target text data, where M is a positive integer greater than 0.

[0108] In an implementation of this embodiment, the device further includes:

[0109] A second acquisition unit is used to acquire text training data of N clean categories; N is a positive integer greater than 0; the category to which the text training data of the clean category obfuscated in the target text data belongs is not greater than N;

[0110] The training unit is used to train the initial text classification model using the text training data of the N clean categories and the cross entropy loss function to obtain the text classification model.

[0111] In an implementation of this embodiment, the device further includes:

[0112] A third acquisition unit is used to acquire text verification data belonging to N clean categories;

[0113] An obtaining unit, used for inputting the text verification data into the text classification model to obtain a clean category prediction result corresponding to the text verification data;

[0114] The updating unit is used to stop updating the text classification model when the clean category prediction result corresponding to the text verification data meets the preset verification condition, so as to obtain the verified text classification model.

[0115] In an implementation of this embodiment, the first cleaning unit 402 includes:

[0116] A first input subunit, used for inputting the text training data of the clean category into the text classification model, obtaining a first statistic of the text training data of the clean category, and calculating a control limit value of the first statistic;

[0117] A second input subunit, used for inputting the target text data into the text classification model to obtain a second statistic of each text data in the target text data under the clean category;

[0118] The cleaning subunit is used to compare the second statistic with the control limit value, and determine whether a text data in the target text data corresponding to the second statistic belongs to the clean category based on the comparison result. If so, clean it from the target text data to obtain the first cleaning result.

[0119] In one implementation of this embodiment, the cleaning subunit is specifically used for:

[0120] It is determined whether the absolute value of the second statistic is less than the absolute value of the control limit value. If so, it means that a text data in the target text data corresponding to the second statistic belongs to the clean category. After cleaning it from the target text data, the first cleaning result is obtained.

[0121] In one implementation of this embodiment, the cleaning subunit includes:

[0122] A calculation subunit, used to calculate the absolute value of the difference between the second statistic corresponding to each text data in the target text data and the control limit value, and determine the target second statistic corresponding to the absolute value less than a preset absolute value threshold;

[0123] The determination subunit is used to determine whether the text data in the target text data corresponding to the target second statistic belongs to a clean category, and obtain a first cleaning result after cleaning it from the target text data.

[0124] In one implementation of the present embodiment, the preset condition for stopping data cleaning is that after two consecutive data cleanings, the total amount of text data belonging to the clean category cleaned from the target text data fluctuates by less than a preset number of items; or the preset condition for stopping data cleaning is that the amount of text data remaining after the target text data is cleaned accounts for no less than a preset proportion of the total amount of original text data of the target text data.

[0125] Furthermore, the embodiment of the present application also provides a text data cleaning device, including: a processor, a memory, and a system bus;

[0126] The processor and the memory are connected via the system bus;

[0127] The memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned text data cleaning method.

[0128] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned text data cleaning method.

[0129] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned text data cleaning method.

[0130] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0131] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0132] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text data cleaning method, characterized in that: include: Acquire target text data to be cleaned; the target text data is mixed with some clean category text data; After training the text classification model using the text training data of the clean category, inputting the target text data and the text training data of the clean category into the text classification model for first data cleaning to obtain a first cleaning result; the first cleaning result includes the text data belonging to the clean category cleaned from the target text data and the target text data remaining after cleaning; Determining whether the first cleaning result meets the preset condition for stopping cleaning data; If yes, the first cleaning result is used as the final cleaning result corresponding to the target text data; If not, the text classification model is trained for a second time using the text training data of the clean category and the target text data remaining after the cleaning to obtain an updated text classification model, and the target text data is cleaned for a second time using the updated text classification model and the text training data of the clean category to obtain a second cleaning result; And so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning data, then the Mth cleaning result is used as the final cleaning result corresponding to the target text data, where M is a positive integer greater than 0; The step of inputting the target text data and the clean category text training data into the text classification model for first data cleaning to obtain a first cleaning result includes: Inputting the text training data of the clean category into the text classification model, obtaining a first statistic of the text training data of the clean category, and calculating a control limit value of the first statistic; Inputting the target text data into the text classification model to obtain a second statistic for each text data in the target text data under the clean category; The second statistic is compared with the control limit value, and based on the comparison result, it is determined whether a text data in the target text data corresponding to the second statistic belongs to the clean category. If so, it is cleaned from the target text data to obtain a first cleaning result.

2. The method according to claim 1, characterized in that The method further comprises: Obtain text training data of N clean categories; N is a positive integer greater than 0; the category to which the obfuscated clean category text data in the target text data belongs is no greater than N; The initial text classification model is trained using the text training data of the N clean categories and the cross entropy loss function to obtain the text classification model.

3. The method according to claim 2, characterized in that The method further comprises: Get text verification data belonging to N clean categories; Inputting the text verification data into the text classification model to obtain a clean category prediction result corresponding to the text verification data; When the clean category prediction result corresponding to the text verification data meets the preset verification condition, the updating of the text classification model is stopped to obtain the verified text classification model.

4. The method according to claim 1, characterized in that: The second statistic is compared with the control limit value, and according to the comparison result, it is determined whether a text data in the target text data corresponding to the second statistic belongs to a clean category, and if so, it is cleaned from the target text data to obtain a first cleaning result, including: It is determined whether the absolute value of the second statistic is less than the absolute value of the control limit value. If so, it means that a text data in the target text data corresponding to the second statistic belongs to the clean category. After cleaning it from the target text data, the first cleaning result is obtained.

5. The method according to claim 1, characterized in that The second statistic is compared with the control limit value, and according to the comparison result, it is determined whether a text data in the target text data corresponding to the second statistic belongs to a clean category, and if so, it is cleaned from the target text data to obtain a first cleaning result, including: Calculate the absolute value of the difference between the second statistic corresponding to each text data in the target text data and the control limit value, and determine the target second statistic corresponding to the absolute value less than a preset absolute value threshold; After determining that the text data in the target text data corresponding to the target second statistic belongs to the clean category and cleaning it from the target text data, a first cleaning result is obtained.

6. The method according to any one of claims 1 to 5, characterized in that: The preset condition for stopping data cleaning is that after two consecutive data cleanings, the total amount of text data belonging to the clean category cleaned from the target text data fluctuates by less than a preset number of items; or the preset condition for stopping data cleaning is that the amount of text data remaining after the target text data is cleaned accounts for no less than a preset proportion of the total amount of original text data of the target text data.

7. A text data cleaning device, characterized in that: include: A first acquisition unit is used to acquire target text data to be cleaned; the target text data is mixed with some clean category text data; A first cleaning unit is used for inputting the target text data and the clean class text training data into the text classification model for first data cleaning after training the text classification model with the clean class text training data to obtain a first cleaning result; the first cleaning result includes the clean class text data cleaned from the target text data and the target text data remaining after cleaning; A second cleaning unit, used to determine whether the first cleaning result meets a preset condition for stopping cleaning data; If yes, the first cleaning result is used as the final cleaning result corresponding to the target text data; If not, the text classification model is trained for a second time using the text training data of the clean category and the target text data remaining after the cleaning to obtain an updated text classification model, and the target text data is cleaned for a second time using the updated text classification model and the text training data of the clean category to obtain a second cleaning result; And so on, until the Mth cleaning result obtained meets the preset condition of stopping cleaning data, then the Mth cleaning result is used as the final cleaning result corresponding to the target text data, where M is a positive integer greater than 0; The first cleaning unit comprises: A first input subunit, used for inputting the text training data of the clean category into the text classification model, obtaining a first statistic of the text training data of the clean category, and calculating a control limit value of the first statistic; A second input subunit, used for inputting the target text data into the text classification model to obtain a second statistic of each text data in the target text data under the clean category; The cleaning subunit is used to compare the second statistic with the control limit value, and determine whether a text data in the target text data corresponding to the second statistic belongs to the clean category based on the comparison result. If so, clean it from the target text data to obtain the first cleaning result.

8. A text data cleaning device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

Images