Data augmentation method and device for multiple types of text

Abstract

The application provides a multi-type text data enhancement method and a text data classification method and device. The method comprises: obtaining a plurality of type text data sets according to type identifiers corresponding to text data; performing word segmentation processing on each piece of text data in each type to obtain a word segmentation set corresponding to the type, and calculating the number of valid words in the word segmentation set of each type; for each type, calculating a type diversity index of the type according to the number of valid words of the type and the total number of text data of the type; selecting at least one to-be-processed type from the plurality of types according to the type diversity index; and performing data enhancement processing on the to-be-processed type. In the case where the text data belongs to a plurality of different types, the type diversity index of each type is calculated, so that data enhancement processing is performed only on a specific type, thereby improving the pertinence of the enhancement processing.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a data enhancement method and apparatus for multiple types of text. Background Technology

[0002] Machine learning models often "overfit" when trained on a limited number of examples. Overfitting occurs when a machine learning model performs accurately on its training examples but fails to generalize to unseen data. Several methods exist in machine learning to avoid overfitting, such as choosing different algorithms, modifying the model's structure, and tuning parameters. Ultimately, the primary method to address overfitting is to add more high-quality data to the training dataset. Sufficient training data can significantly improve the predictive performance of machine learning models. However, obtaining high-quality data remains a significant challenge.

[0003] To address the aforementioned issues, related technologies employ "data augmentation" operations on training data, which generate new training samples from existing ones. Data augmentation is a low-cost and effective method for improving the performance and accuracy of machine learning models under data constraints. Existing data augmentation methods typically augment the entire dataset, performing well with datasets of a single type. However, if the dataset contains multiple data types, existing data augmentation methods cannot effectively augment the data across these diverse types. Summary of the Invention

[0004] This application provides a data augmentation method, a text data classification method, and an apparatus for augmenting multiple types of text data.

[0005] Firstly, this application provides a data augmentation method for multiple types of text, the method comprising:

[0006] Based on the type identifier corresponding to the text data, obtain text datasets of multiple types;

[0007] For each type, word segmentation is performed on each text data in the text dataset of that type to obtain the word segmentation set corresponding to that type. The number of valid word segments contained in the word segmentation set of each type is calculated; wherein, the number of valid word segments is used to characterize the number of different word segments contained in the word segmentation set of that type.

[0008] For each type, a type diversity index is calculated based on the number of effective word segments for that type and the total number of text data entries for that type.

[0009] Based on the type diversity index, at least one type to be processed is determined from multiple types;

[0010] Perform data augmentation processing on the text data corresponding to the type to be processed.

[0011] Secondly, this application provides a data enhancement device for multiple types of text, including:

[0012] The acquisition module is suitable for acquiring text datasets of multiple types based on the type identifiers corresponding to the text data;

[0013] The word segmentation module is adapted to perform word segmentation processing on each text data in the text dataset of each type, to obtain the word segmentation set corresponding to the type, and to calculate the number of effective word segments contained in the word segmentation set of each type; wherein, the number of effective word segments is used to characterize the number of different word segments contained in the word segmentation set of the type;

[0014] The calculation module is adapted to calculate the type diversity index for each type based on the number of effective word segments for that type and the total number of text data entries for that type.

[0015] The filtering module is adapted to determine at least one type to be processed from multiple types based on the type diversity index;

[0016] The enhancement module is adapted to perform data enhancement processing on the text data corresponding to the type to be processed.

[0017] Thirdly, this application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0018] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described method when executed by a processor / processor core.

[0019] The embodiments provided in this application can perform word segmentation processing on each piece of text data in each type, obtain a word segmentation set corresponding to the type, and calculate the number of effective words contained in the word segmentation set of each type. Furthermore, for each type, based on the number of effective words in the type and the total number of text data entries in the type, a type diversity index is calculated, thereby selecting at least one type from multiple types as the type to be processed, so that data augmentation processing can be performed on the type to be processed. Therefore, this method can obtain the type diversity index for each type based on the comparison between the number of effective words in each type and the total number of text data entries in that type, thereby selecting at least one type with insufficient diversity from multiple types, and performing data augmentation on the type with insufficient diversity without needing to augment data types with sufficient diversity. Thus, this method can calculate the type diversity index for each type separately when the text data belongs to multiple different types, thereby performing data augmentation processing only on specific types, improving the targeting of the augmentation processing.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed example embodiments described with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart illustrating a data augmentation method for multiple text types provided in one embodiment of this application;

[0023] Figure 2 A flowchart illustrating a data augmentation method for multiple text types provided in another embodiment of this application;

[0024] Figure 3 A flowchart illustrating a text data classification method is shown.

[0025] Figure 4 A block diagram of a data enhancement device for multiple types of text provided in an embodiment of this application is shown;

[0026] Figure 5 A schematic diagram of the structure of a text data classification device provided in another embodiment of this application;

[0027] Figure 6This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below in conjunction with the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.

[0030] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0032] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0033] The data enhancement method for multiple types of text according to embodiments of this application can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Specifically, the method can be implemented by a processor calling a computer program stored in memory.

[0034] In related technologies, augmentation is typically performed on the entire dataset, which performs well with datasets of a single type. However, if the dataset contains multiple types of data, existing data augmentation methods cannot perform effective augmentation for these diverse data types. The greater the diversity of text data included in the training samples, the higher the classification accuracy of the final trained classification model. To address this issue, this application proposes a data augmentation scheme based on type diversity indices. By calculating the type diversity index for each type separately, targeted processing is achieved for types with insufficient diversity.

[0035] Figure 1 A flowchart of a data augmentation method for multiple text types, according to an embodiment of this application, is provided. (Refer to...) Figure 1 The method includes:

[0036] Step S110: Obtain text datasets of multiple types based on the type identifiers corresponding to the text data.

[0037] The type identifier corresponding to the text data can take various forms, such as type label or type code, and is used to uniquely identify the type of a text data. In this embodiment, multiple types of text datasets are obtained, and each type of text dataset contains multiple pieces of text data belonging to the same type.

[0038] Step S120: For each type, perform word segmentation on each text data in the text dataset of the type to obtain the word segmentation set corresponding to the type, and calculate the number of valid words contained in the word segmentation set of each type; wherein, the number of valid words is used to represent the number of different words contained in the word segmentation set of the type.

[0039] For each type, each piece of text data within that type is segmented into words. The corresponding word segmentation set for that type contains the segmentation results for each piece of text data in that type. A deduplication operation is then performed on the multiple word segments contained in the word segmentation set for each type, yielding the number of valid word segments in that type's word segmentation set. Therefore, the number of valid word segments represents the number of distinct word segments contained in the word segmentation set for a given type. For example, assuming a word segmentation set for a type contains a total of 50 word segments, and 20 of them are duplicates, the number of valid word segments after deduplication is 30. In short, the number of valid word segments refers to the number of word segments obtained after performing a deduplication operation on the word segments contained in the word segmentation set for a given type.

[0040] Step S130: For each type, calculate the type diversity index based on the number of effective word segments for the type and the total number of text data entries for the corresponding type.

[0041] The total number of text data entries for each type represents the total amount of text data contained in that type. Correspondingly, the comparison between the number of effective word segments for a type and the total number of text data entries for that type can characterize the data diversity of that type. For example, if the total number of text data entries for a type is smaller and the number of effective word segments is larger, it indicates that a single text data entry contains more effective content, and therefore has higher diversity.

[0042] Step S140: Based on the type diversity index, identify at least one type to be processed from multiple types.

[0043] Since type diversity indicators can characterize the data diversity of a corresponding type, types with insufficient diversity are selected from multiple types as the types to be processed. Specifically, this can be done by setting indicator thresholds, or by sorting multiple type diversity indicators; this invention does not limit the specific details.

[0044] Step S150: Perform data augmentation processing on the text data corresponding to the type to be processed.

[0045] Data augmentation can be implemented in various ways. For example, the total amount of data can be increased to expand the data type being processed, thereby enhancing its diversity. Alternatively, the total amount of data can be reduced to decrease the data type, also improving its diversity. Furthermore, data augmentation can be performed on specific data within the data type being processed; for instance, data expansion can be performed on data with good diversity, while data reduction can be performed on data with poor diversity. In summary, this invention does not limit the specific implementation of data augmentation. This solution aims to identify types with insufficient diversity and perform targeted augmentation on these types to improve the effectiveness of the augmentation.

[0046] In the embodiments provided in this application, a type diversity index for each type can be obtained by comparing the number of effective word segments for each type with the total number of text data entries for that type. This allows for the selection of at least one type with insufficient diversity from multiple types, enabling data augmentation for the type with insufficient diversity without needing to augment data types with sufficient diversity. Therefore, this method can calculate the type diversity index for each type separately when the text data belongs to multiple different types, thereby performing data augmentation processing only on specific types and improving the targeting of the augmentation process.

[0047] See Figure 2 The flowchart below illustrates another data augmentation method for multi-type text provided in this application. (See attached diagram.) Figure 2 The method may include the following steps:

[0048] Step S210: Obtain text datasets of multiple types based on the type identifiers corresponding to the text data.

[0049] The type identifier corresponding to the text data can be in various forms such as type label or type code, used to uniquely identify the type of a piece of text data. In this embodiment, multiple types of text data are obtained.

[0050] Step S220: For each type, perform word segmentation on each piece of text data in that type to obtain the word segmentation set corresponding to that type.

[0051] Specifically, for each type, each piece of text data in that type is segmented into words. Correspondingly, the word segmentation set for that type contains the word segmentation results for each piece of text data in that type.

[0052] Step S230: Calculate the number of valid segments in each type of segmentation set, and obtain the total number of segments, the average text length of the text data, and the average word length of the segments for that type; calculate the type diversity index for each type based on the number of valid segments, the total number of segments, the average text length of the text data, and the average word length of the segments.

[0053] The effective word segmentation count is used to characterize the number of different words contained in the word segmentation set of a type. Therefore, this step aims to calculate the type diversity index based on the effective word segmentation count of the type and the total number of text data entries for that type. In practice, this can be achieved as follows:

[0054] First, for each type, obtain the total number of segmented words, the average text length of the text data, and the average word length of the segmented words. The total number of segmented words for that type represents the total number of segmented words in the segmented word set. The average text length of the text data for that type represents the average text length of multiple text data entries. The average word length of the segmented words for that type represents the average length of multiple segmented words. Then, calculate a first ratio between the effective number of segmented words for that type and the total number of segmented words for that type, and a second ratio between the average text length of the text data for that type and the average word length of the segmented words. Finally, based on the first ratio, the second ratio, and the total number of text data entries for that type, calculate the type diversity index for that type.

[0055] In one specific implementation, the data shown in Table 1 is obtained through data statistics:

[0056] Table 1

[0057] class_all N n M L l class_1 <![CDATA[N1]]> <![CDATA[n1]]> <![CDATA[M1]]> <![CDATA[L1]]> <![CDATA[l1]]> class_2 <![CDATA[N2]]> <![CDATA[n2]]> <![CDATA[M2]]> <![CDATA[L2]]> <![CDATA[l2]]> … … … … … … class_m <![CDATA[N m ]]> <![CDATA[n m ]]> <![CDATA[M m ]]> <![CDATA[L m ]]> <![CDATA[l m ]]>

[0058] In the example shown in Table 1, it is assumed that the original labeled data has a total of m categories.

[0059] First, data statistics were performed. The results are shown in Table 1.

[0060] Category: class_all represents the sum of all categories, that is, all data in the entire labeled dataset; class_1, class_2, ..., class_m represent category 1, category 2, ..., category m, respectively.

[0061] Total word count: N represents the total number of words contained in the entire labeled data set; N1, N2, ..., N m These represent the total number of words (i.e., the total number of words in each category) in category 1, category 2, ..., category m, respectively; N = N1 + N2 + ... + N m .

[0062] Different word counts (i.e., effective word segmentation count): n represents the number of distinct words in the total number of words N contained in the labeled dataset; n1, n2, ..., n m N1, N2, ..., Nm represent the total number of words contained in category 1, category 2, ..., category m, respectively. m How many different words are there in it?

[0063] Text count: M represents the number of text entries contained in the entire labeled data set; M1, M2, ..., M m These represent the number of text entries contained in class_1, class_2, ..., class_m (i.e., the total number of text data entries of each type); M = M1 + M2 + ... + M m .

[0064] Average text length: L represents the average text length of the entire labeled dataset; L1, L2, ..., L m These represent the average text length (i.e., the average text length of the text data) for class_1, class_2, ..., class_m, respectively; the average text length is equal to the total number of characters in that class divided by the number of text entries in that class.

[0065] Average word length: l represents the average word length of the entire labeled dataset; l1, l2, ..., l m These represent the average word length (i.e., the average word length of word segmentation) for class_1, class_2, ..., class_m, respectively; the average word length is equal to the total number of characters in the class divided by the total number of words in the class.

[0066] Then, overall diversity is calculated to set an overall diversity metric. The overall diversity metric measures the data diversity of the entire labeled dataset. The formula for calculating overall diversity τ is:

[0067]

[0068] in, Used to measure word richness; the higher the value, the greater the word richness and diversity. τ is related to... Proportional; τ represents the average number of words per sentence. A higher value indicates a greater number of words per sentence and richer word diversity. It is directly proportional; when the number of words and the average number of words per sentence are constant, τ is proportional to... It is inversely proportional, meaning that the fewer the sentences, the richer the diversity.

[0069]

[0070] Substituting formula (2) into formula (1), we get:

[0071]

[0072] Therefore, the overall diversity index τ is directly proportional to the number of different words n in the total number of words N in the annotated data set, and inversely proportional to the square of the number of text entries M in the annotated data set.

[0073] The type diversity metric for each type measures the data diversity of each category in the labeled data. For example, the diversity τ of the i-th category, class_i. class_i The calculation formula is:

[0074]

[0075] Where, n i This indicates how many distinct words constitute category_i, i.e., the effective word segmentation count of category_i; M i This indicates the number of text entries contained in category class_i.

[0076] Step S240: Compare the type diversity index of each type with the overall diversity index; select types whose type diversity index is not greater than the overall diversity index as types to be processed.

[0077] The overall diversity index is determined based on the sum of the number of valid word segments in multiple types of word segmentation sets and the sum of the total number of text data entries across multiple types. For example, the overall diversity index τ is determined according to the formula above. The diversity τ of the i-th (1≤i≤m) class_i is... class_i The criteria for determining whether the requirements are met are as follows:

[0078] τ class_i >τ

[0079] If τ class_i If the judgment criteria are met, it means that the diversity of class_i in the original data is sufficient and no additional data is needed; otherwise, it means that the diversity of class_i is insufficient and the diversity of this category needs to be enriched by adding data.

[0080] Step S250: Perform data augmentation processing on the type to be processed.

[0081] Data augmentation is used to enrich the diversity of data. For categories lacking diversity, the following methods are used to increase inter-class diversity:

[0082] In the first implementation, for the text data in the type to be processed, data expansion is performed by matching a thesaurus; and / or, for the text data in the type to be processed, the similarity between each pair of text data is calculated, and text data with a similarity greater than a preset value is deleted.

[0083] The method of matching synonym vocabularies is used to increase word richness. In this embodiment, we take the j-th category, class_j, as an example, assuming that class_j has insufficient diversity. First, we use n from class_j... j Each of the following words is matched against a thesaurus to obtain synonyms for each word. The matching rule is: n j The synonyms matched by each of the different words are distinct from each other, and the matched synonyms are not related to n. j The word is repeated. Then, use n j The system finds synonyms for each different word and replaces the corresponding word, thus increasing the richness of the vocabulary.

[0084] Additionally, text deletion based on similarity is used to reduce the number of sentences in class_j. First, duplicate text is removed to reduce the number of sentences. Then, the similarity between any two sentences in class_j is measured using the Jaccard similarity coefficient. For sentence pairs with similarity values ​​greater than a certain threshold, they are sorted from largest to smallest similarity value, and one sentence from each pair is removed based on the sorting result. The removal priority is set as follows: a. Delete the sentence with fewer characters; b. If the number of characters is the same, delete the sentence with more words; c. If the number of words is the same, delete the sentence that appears most frequently among all sentence pairs with similarity values ​​greater than a certain threshold; d. If the frequency of occurrence is the same, delete one sentence randomly. The total number of deleted sentences should not exceed a preset proportion of the number of sentences in class_j; for example, the total number of deleted sentences should not exceed 1 / 5 of the number of sentences in class_j.

[0085] In the second implementation, the text data is expanded through retrieval operations:

[0086] First, each pair of text data in the type to be processed is treated as a text pair, and the similarity between the two text data in each text pair is determined as the text pair similarity of the text pair.

[0087] Then, based on text pair similarity, multiple text pairs are divided into at least two threshold groups; the text pair similarity is different in different threshold groups. For example, the text similarity between any two texts in category clss_j is calculated, and the text pair is divided into different threshold groups according to the magnitude of the text similarity value. There are four threshold groups: (α1, α2], (α2, α3], (α3, α4], and (α4, α5), where α1, α2, α3, α4, and α5 can take values ​​of 0, 0.55, 0.70, 0.85, and 1, respectively.

[0088] Next, for each threshold group, the average text vector of the text pairs in the threshold group is calculated. Scene data with a similarity to the average text vector less than a preset similarity threshold is retrieved from a preset scene database. The text data in the threshold group is then expanded based on the retrieved scene data. The similarity threshold for each threshold group is determined based on the text pair similarity among the text pairs contained in the threshold group.

[0089] The average text vector of text pairs in each threshold group can be calculated using either of the following two methods:

[0090] The first calculation method: Taking the (α1, α2] threshold group as an example, firstly, remove the texts that overlap with the other three threshold groups from the text in the (α1, α2] threshold group, and then calculate the average vector of the remaining texts in the (α1, α2] threshold group. The average vector is calculated by adding the vectors of each text and dividing by the number of texts.

[0091] The second calculation method: Taking the (α1, α2] threshold group as an example, firstly, count the number of times each text in the (α1, α2] threshold group appears in the four threshold groups. If a text appears in all four threshold groups, the count is 4; if it appears in three threshold groups, the count is 3; if it appears in two threshold groups, the count is 2; and if it appears in one threshold group, the count is 1. Then, based on the number of times each text in the (α1, α2] threshold group appears in the four threshold groups, assign different weights to each text. The counts of 1, 2, 3, and 4 correspond to... The weights are β1, β2, β3, and β4, which can take values ​​of 0.6, 0.2, 0.15, and 0.05 respectively. The average vector of the texts in the (α1, α2) threshold group is calculated as: each text multiplied by its corresponding weight, summed, and then divided by the number of texts. This calculation method considers the following: if a text appears in multiple threshold groups, its distinguishability is low, indicating that its features are not distinct enough, and therefore a lower weight is assigned to it; conversely, if a text appears only in one threshold group, its distinguishability is high, indicating that its features are distinct, and therefore a higher weight is assigned to it.

[0092] Specifically, when retrieving scene data from a preset scene database that has a similarity to the average text vector less than a preset similarity threshold, and expanding the text data in the threshold group based on the retrieved scene data, this is achieved in the following way:

[0093] First, text retrieval is performed: text retrieval is conducted in the scene database containing the labeled dataset to enrich the data diversity of this category. The average text vectors corresponding to the four threshold groups obtained above are used to calculate the similarity between a text in the scene database and the four average text vectors. The condition for determining whether a text can be added to this category is:

[0094] a. If a text satisfies the similarity threshold μ with one of the four average text vectors, then the text is added to the category; if a text satisfies the similarity threshold μ with multiple (two or more) of the four average text vectors, or if the similarity between a text and all four average text vectors is less than μ, then the text is not added to the category. The similarity threshold μ can be 0.9.

[0095] b. Within this category, the number of texts required to be added for each of the four threshold groups is different, with the ratio of text additions being γ1:γ2:γ3:γ4. The values ​​of γ1, γ2, γ3, and γ4 can be 0.4, 0.3, 0.2, and 0.1, respectively. All of the above conditions must be met simultaneously for a text to be added to this category. For example, if this category needs 100 more texts to enrich its diversity, according to condition b, the four threshold groups need to add 40, 30, 20, and 10 texts, respectively. Combining this with condition a, that is, while a text satisfies condition a (the text meets the similarity threshold condition with one of the four average text vectors), the number of texts required to be added for the threshold group corresponding to that average text vector (the average text vector that meets the similarity threshold condition with the text) must not meet the requirement.

[0096] Optionally, after performing data augmentation on the text data corresponding to the type to be processed, the following operations are performed: a text classification model is trained based on the text datasets of multiple types after data augmentation; the text data to be classified is input into the text classification model; the data type of the text data to be classified is determined based on the output of the text classification model; wherein the data length of the text data to be classified is less than a preset length threshold.

[0097] To facilitate understanding, a specific application scenario of this embodiment will be described in detail below using a concrete example:

[0098] In related technologies, data diversity is receiving increasing attention in machine learning, becoming a major challenge for machine learning models in practical deployment. When the diversity distribution of data does not cover the actual scenario, there is a significant deviation between the diversity distribution of the data to be predicted and the diversity distribution of the data used for training, leading to poor model performance. This poor performance caused by data diversity is difficult to resolve by improving the model's generalization ability, as current machine learning methods are primarily based on the premise of independent and identically distributed (IOD) data. While the observable training data under a real distribution is limited, a well-trained model will experience a drop in accuracy when encountering unobserved samples that conform to the same distribution during prediction. In such cases, choosing appropriate algorithms, cross-validation, and regularization can effectively improve the model's generalization ability. However, the essence of the data diversity problem is that the real data distribution differs significantly from the actual scenario distribution; therefore, simply improving generalization ability is insufficient to effectively improve model performance. Currently, industry and academia address these challenges through methods such as: manually constructing, screening, and labeling samples, which is effective but requires significant human and material resources and specialized expertise; and data augmentation, which is simple and easy to implement, solving the resource constraints, but can introduce noise and negatively impact model performance if used improperly. These methods all aim to improve the model's generalization ability by increasing the diversity of training data. However, these methods also have significant drawbacks: they lack scientific analysis of the original data. Some categories in the original data have sufficient diversity and do not require additional data, while others lack diversity and need enrichment. Blindly augmenting data globally wastes resources (time, machine costs, etc.) and can even be counterproductive. Therefore, this example proposes a data diversity detection method to identify which categories in the data lack diversity and then selectively enrich their diversity to improve model performance.

[0099] This example primarily applies to the knowledge base construction process of a chatbot. The knowledge base contains multiple categories (for example, in a banking chatbot, the knowledge base includes categories such as "credit processing," "credit card application," "points redemption," and "bank card cancellation"). The text in each category consists of standard questions and similar questions (for example, in "credit card application," standard questions could be: "What are the standards for credit card application?" or "How do I apply for a credit card?"; similar questions could be: "I want to apply for a credit card, what is the process?" or "Can someone under 18 apply for a credit card?"). Standard questions are generally formulated by business experts, while similar questions are mined from historical customer data. The data diversity of the knowledge base determines the service quality of the chatbot; therefore, it is necessary to test the overall diversity of the knowledge base and the data diversity of each category. In the chatbot's knowledge base, the text in each category (standard questions and similar questions) has the following characteristics: short text; colloquial language; and only a few words used to express intent. In this scenario, text representation methods (clustering, similarity calculation, etc.) cannot objectively reflect the diversity of the data. This invention measures data diversity from the perspective of text composition. It statistically analyzes the overall labeled data and text data of each category, including the total number of words, the number of different words, the number of text entries, the average text length, and the average word length. Based on the above statistical data, a diversity calculation method is designed to calculate the overall diversity and inter-category diversity of the labeled data. Through diversity judgment rules, categories with insufficient inter-category diversity are identified. Two different schemes are designed for categories with insufficient diversity to improve inter-category diversity.

[0100] The characteristics of this application scenario are: text representation methods cannot accurately reflect text type, while statistical methods can better reflect text type, eliminating the need to consider semantics and focusing only on character similarity. For example, "I like playing ball" and "I don't like playing ball" are similar, but their semantics are clearly different. In this scenario, only a few keywords in the user's speech are needed to express their intent.

[0101] Optionally, in another specific example, the diversity detection method is applied to the text sentiment recognition scenario in the dialogue domain. Specifically, sentiment recognition needs to be performed on the spoken text of both parties in the dialogue. Typically, coarse sentiment classification includes three types: positive, neutral, and negative, while fine sentiment classification includes multiple types such as no emotion, liking, sadness, disgust, anger, and happiness. The text in this scenario has the following characteristics: the text is relatively short; it is more colloquial; and only a few words in the text are used to distinguish emotions. The diversity of the labeled data for each emotion determines the recognition accuracy and the strength of generalization for that emotion type. Therefore, it is necessary to detect the overall diversity of the sentiment labeled data and the data diversity of each type. For example, sentiment text of the "sad" type includes: "Life is short, I'm so sad.", "Yes, it's a pity I haven't been there.", "Yes! I'm heartbroken too.", etc.; sentiment text of the "happy" type includes: "Yes, hahaha...", "The weather is so nice! The sky is clear!", "This is the ancient city last night, it's very beautiful.", etc. As can be seen, in this example, the text is short and only a few words are used to represent emotional tone. Therefore, the statistical word segmentation method proposed in this scheme helps to improve the effect of diversity detection.

[0102] The present invention can achieve at least the following beneficial effects:

[0103] This invention proposes a method for data diversity detection. By statistically analyzing the overall labeled data and text data of each category, designing methods for calculating overall diversity and inter-class diversity, establishing diversity judgment rules, and implementing diversity enrichment methods, it detects sample diversity and improves the model's generalization ability. Specifically, the diversity calculation method provides a novel way to measure sample diversity, enabling rapid and effective calculation of overall and inter-class diversity, avoiding blind model training. The diversity judgment rules can identify categories with insufficient inter-class diversity before model training, avoiding inferring inter-class diversity based on model performance after training. The diversity enrichment scheme provides a scientific method for enriching inter-class diversity, improving the model's generalization ability and solving the problem of blindly expanding data globally, which wastes resources (time cost, machine cost, etc.) and can even have adverse effects (easily introduces noise, causing data pollution; the semantics of expanded data are difficult to control; and it cannot achieve global optimization). Furthermore, this scheme features high computational speed and a high degree of automation.

[0104] In addition, another embodiment of the present invention provides a method for classifying text data. Figure 3 A flowchart illustrating this classification method is shown, as follows: Figure 3 As shown, the method includes:

[0105] Step S310: Input the text data to be classified into the text classification model.

[0106] Step S320: Determine the data type of the text data to be classified based on the output of the text classification model.

[0107] Among them, the text classification model is achieved by... Figure 1 or Figure 2 The method shown is used to train multiple types of text data after data augmentation; and the length of the text data to be classified is less than a preset length threshold.

[0108] This text classification method allows for pre-processing of data augmentation for specific types in the training data, thereby increasing the diversity of data types that lack diversity. This results in more accurate output from the text classification model, leading to more accurate text classification results.

[0109] It is understood that the various method embodiments mentioned above in this application can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this application will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0110] In addition, this application also provides a data enhancement device and electronic device for multiple types of text, and a computer-readable storage medium. All of the above can be used to implement any of the data enhancement methods for multiple types of text provided in this application. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0111] Figure 4 This is a schematic diagram of the structure of a data enhancement device 40 for multiple types of text provided in an embodiment of this application.

[0112] Reference Figure 4 This application provides a data enhancement device 40 for multiple types of text, which includes:

[0113] The acquisition module 41 is adapted to acquire multiple types of text data based on the type identifier corresponding to the text data;

[0114] The word segmentation module 42 is adapted to perform word segmentation processing on each piece of text data in each type to obtain a word segmentation set corresponding to the type, and calculate the number of effective word segments contained in the word segmentation set of each type; wherein, the number of effective word segments is used to characterize the number of different word segments contained in the word segmentation set of the type;

[0115] Calculation module 43 is adapted to calculate the type diversity index for each type based on the number of effective word segments of the type and the total number of text data entries of the type;

[0116] The filtering module 44 is adapted to filter at least one type from multiple types as the type to be processed based on the type diversity index.

[0117] Enhancement module 45 is adapted to perform data enhancement processing for the type to be processed.

[0118] Optionally, the word segmentation module is specifically adapted to:

[0119] For each type, obtain the total number of words segmented, the average text length of the text data, and the average word length of the segmented words for that type;

[0120] Calculate a first ratio between the number of effective word segments of the type and the total number of word segments of the type, and a second ratio between the average text length of the text data of the type and the average word length of the word segments;

[0121] The type diversity index of the type is calculated based on the first ratio, the second ratio, and the total number of text data of the type.

[0122] Optionally, the plurality of types is m types, and the calculation module is specifically adapted to:

[0123] For the i-th type, the type diversity index is calculated using the following formula:

[0124]

[0125] Where, τ class_i Let n be the diversity index of the i-th type. i M represents the number of valid word segments of the i-th type; i This represents the total number of text data entries of the i-th type.

[0126] Optionally, the filtering module is specifically adapted to:

[0127] Compare the type diversity index for each type with the overall diversity index;

[0128] Types whose type diversity index is not greater than the overall diversity index are selected as types to be processed;

[0129] The overall diversity index is determined based on the sum of the number of valid word segments contained in multiple types of word segmentation sets and the sum of the total number of text data entries of multiple types.

[0130] Optionally, the enhancement module is specifically adapted to:

[0131] For the text data of the aforementioned type to be processed, data expansion processing is performed by matching a thesaurus; and / or,

[0132] For the text data in the type to be processed, calculate the similarity between each pair of text data, and delete the text data with a similarity greater than a preset value.

[0133] Optionally, the enhancement module is specifically adapted to:

[0134] Each pair of text data in the type to be processed is taken as a text pair, and the similarity between the two text data in each text pair is determined as the text pair similarity of the text pair.

[0135] Based on text pair similarity, multiple text pairs are divided into at least two threshold groups; wherein, the text pair similarity of text pairs in different threshold groups is different;

[0136] For each threshold group, the average text vector of the text pairs in the threshold group is calculated, and scene data with a similarity to the average text vector less than a preset similarity threshold is retrieved from a preset scene database. The text data in the threshold group is then expanded based on the retrieved scene data.

[0137] The similarity threshold for each threshold group is determined based on the text pair similarity among the text pairs contained in the threshold group.

[0138] Figure 5 This is a schematic diagram of the structure of a text data classification device 50 provided in another embodiment of this application. (See attached diagram.) Figure 5 As shown, the text data classification device includes:

[0139] Input module 51 is adapted to input text data to be classified into a text classification model;

[0140] The classification module 52 is adapted to determine the data type of the text data to be classified based on the output of the text classification model.

[0141] The text classification model is trained using multiple types of text data after data augmentation processing as described above; and the length of the text data to be classified is less than a preset length threshold.

[0142] Figure 6 This is a block diagram of an electronic device provided in an embodiment of this application.

[0143] Reference Figure 6This application provides an electronic device, which includes: at least one processor 501; at least one memory 502; and one or more I / O interfaces 503 connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to perform the above-described data enhancement method for multiple types of text.

[0144] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the aforementioned data enhancement method for multiple types of text. The computer-readable storage medium can be volatile or non-volatile.

[0145] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described data enhancement method for multiple types of text.

[0146] Those skilled in the art will understand that all or some of the steps, systems, or devices disclosed above, as well as their functional modules / units, can be implemented as software, firmware, hardware, or other similar components.

[0147] Appropriate combinations. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0148] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0149] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0150] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0151] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0152] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0153] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0154] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0156] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. A data augmentation method for multi-type text, characterized in that, include: Based on the type identifier corresponding to the text data, obtain text datasets of multiple types; For each type, word segmentation is performed on each text data in the text dataset of that type to obtain the word segmentation set corresponding to that type. The number of valid word segments contained in the word segmentation set of each type is calculated; wherein, the number of valid word segments is used to characterize the number of different word segments contained in the word segmentation set of that type. For each type, a type diversity index is calculated based on the number of effective word segments for that type and the total number of text data entries for that type. Based on the type diversity index and the overall diversity index, at least one type to be processed is determined from multiple types; the type diversity index of the type to be processed is not greater than the overall diversity index; the overall diversity index is determined based on the sum of the number of effective words contained in the word segmentation sets of multiple types and the sum of the total number of text data entries of multiple types. Retrieve scene data from a preset scene database that has a similarity to the text data corresponding to the type to be processed that is less than a preset similarity threshold, and perform data augmentation processing on the text data corresponding to the type to be processed.

2. The method according to claim 1, characterized in that, For each type, the type diversity index is calculated based on the number of effective word segments in the text dataset of that type and the total number of text data entries of the corresponding type, including: For each type, obtain the total number of words segmented, the average text length of the text data, and the average word length of the segmented words for that type; Calculate a first ratio between the number of effective word segments of the type and the total number of word segments of the type, and a second ratio between the average text length of the text data of the type and the average word length of the word segments; The type diversity index of the type is calculated based on the first ratio, the second ratio, and the total number of text data of the corresponding type.

3. The method according to claim 1 or 2, characterized in that, The multiple types are m types. For each type, the type diversity index is calculated based on the number of effective word segments and the total number of text data entries of the corresponding type, including: For the i-th type, the type diversity index is calculated using the following formula: ;in, , 1≤i≤m; in, For the i-th type of diversity index, This represents the number of valid word segments of the i-th type; This represents the total number of text data entries of the i-th type. This represents the total number of words in the i-th category; This represents the average word length of the i-th type; This represents the average text length of the i-th type.

4. The method according to claim 3, characterized in that, The step of determining at least one type to be processed from multiple types based on the type diversity index and the overall diversity index includes: Compare the type diversity index for each type with the overall diversity index; Types whose type diversity index is not greater than the overall diversity index are selected as types to be processed.

5. The method according to any one of claims 1-2 and 4, characterized in that, The data augmentation process performed on the text data corresponding to the type to be processed includes: For the text data of the aforementioned type to be processed, data expansion processing is performed by matching a thesaurus; and / or, For the text data in the type to be processed, calculate the similarity between each pair of text data, and delete the text data with a similarity greater than a preset value.

6. The method according to any one of claims 1-2 and 4, characterized in that, The step of retrieving scene data from a preset scene database whose similarity to the text data corresponding to the type to be processed is less than a preset similarity threshold, and then performing data augmentation processing on the text data corresponding to the type to be processed, includes: Each pair of text data in the type to be processed is taken as a text pair, and the similarity between the two text data in each text pair is determined as the text pair similarity of the text pair. Based on text pair similarity, multiple text pairs are divided into at least two threshold groups; wherein, the text pair similarity of text pairs in different threshold groups is different; For each threshold group, the average text vector of the text pairs in the threshold group is calculated, and scene data with a similarity to the average text vector less than a preset similarity threshold is retrieved from a preset scene database. The text data in the threshold group is then expanded based on the retrieved scene data. The similarity threshold for each threshold group is determined based on the text pair similarity among the text pairs contained in the threshold group.

7. The method according to any one of claims 1-2 and 4, characterized in that, After performing data augmentation processing on the text data corresponding to the type to be processed, the process further includes: Train a text classification model based on multiple types of text datasets after performing data augmentation; Input the text data to be classified into the text classification model; Based on the output of the text classification model, the data type of the text data to be classified is determined; wherein the length of the text data to be classified is less than a preset length threshold.

8. A data enhancement device for multiple types of text, characterized in that, include: The acquisition module is suitable for acquiring text datasets of multiple types based on the type identifiers corresponding to the text data; The word segmentation module is adapted to perform word segmentation processing on each text data in the text dataset of each type, to obtain the word segmentation set corresponding to the type, and to calculate the number of effective word segments contained in the word segmentation set of each type; wherein, the number of effective word segments is used to characterize the number of different word segments contained in the word segmentation set of the type; The calculation module is adapted to calculate the type diversity index for each type based on the number of effective word segments for that type and the total number of text data entries for that type. The filtering module is adapted to determine at least one type to be processed from multiple types based on the type diversity index and the overall diversity index; the type diversity index of the type to be processed is not greater than the overall diversity index; the overall diversity index is determined based on the sum of the number of effective word segments contained in the word segmentation sets of multiple types and the sum of the total number of text data entries of multiple types. The enhancement module is adapted to retrieve scene data in a preset scene database whose similarity to the text data corresponding to the type to be processed is less than a preset similarity threshold, so as to perform data enhancement processing on the text data corresponding to the type to be processed.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-7.

Images