Text data processing method and device, electronic equipment and storage medium

Abstract

The application provides a text data processing method and device, electronic equipment and storage medium. The method comprises the following steps: classifying a plurality of text data to obtain a category to which each text data belongs; calculating the text similarity between each two text data in each category; determining the diversity evaluation result of each category according to the text similarity between each two text data in each category; determining the category whose diversity evaluation result meets the preset enhancement condition as a to-be-enhanced category, and performing data enhancement processing on the text data in the to-be-enhanced category. The method analyzes the part of text data that needs data enhancement and processes it, avoids the resource waste problem caused by blind global data expansion, and improves the generalization ability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a text data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Data diversity problems have been increasingly concerned in the field of machine learning in recent years. When the diversity distribution of data does not reach the coverage rate of the actual scene, the predicted data diversity distribution and the data diversity distribution used for training show obvious deviation, which will result in poor performance of the model.

[0003] In the related art, it is difficult to solve the problem of poor performance of the model caused by the data diversity problem by improving the generalization ability of the model. Therefore, the main methods to solve the above problems include the following two: manually constructing, screening and labeling samples. This method is effective, but it requires a lot of manpower and resources, and needs to be completed by professionals in the field; data augmentation. This method is simple and easy to operate, and can solve the problem of high cost of manpower and resources, but if not used properly, it can introduce noise and affect the effect of the model. The purpose of the above two methods is to increase the diversity of training data to improve the generalization ability of the model, but neither of them analyzes the original data scientifically. In the original data, the diversity of some categories is sufficient and does not need to be increased; the diversity of some categories is not enough and needs to be enriched. If blind global data augmentation is performed, it will waste resources (time cost, machine cost, etc.), and even have the opposite effect. SUMMARY

[0004] The present application provides a text data processing method and device, electronic equipment and storage medium, which is used to detect the diversity of text data and improve the generalization ability of the model through data augmentation, thereby reducing the waste of resources caused by blind global data augmentation.

[0005] In a first aspect, the present application provides a text data processing method, comprising:

[0006] Classifying a plurality of text data to obtain the category to which each text data belongs;

[0007] For each category, the text similarity between each two text data in the category is calculated respectively;

[0008] According to the text similarity between each two text data in each category, a diversity evaluation result of each category is determined;

[0009] The category whose diversity evaluation result meets the preset augmentation condition is determined as a to-be-augmented category, and data augmentation processing is performed on the text data in the to-be-augmented category.

[0010] In a second aspect, the present application provides a text data processing apparatus, comprising:

[0011] a text classification module configured to classify the plurality of text data to obtain a category to which each text data belongs;

[0012] a similarity calculation module configured to calculate, for each category, a text similarity between each two text data in the category;

[0013] a diversity evaluation module configured to determine a diversity evaluation result of each category according to the text similarity between each two text data in the category;

[0014] a data enhancement module configured to determine, as a to-be-enhanced category, a category whose diversity evaluation result meets a preset enhancement condition, and perform data enhancement processing on the text data in the to-be-enhanced category.

[0015] In a third aspect, the present application provides an electronic device, comprising: at least one processor; a memory connected to the at least one processor in communication; and one or more I / O interfaces connected between the at least one processor and the memory; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the text data processing method described above.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text data processing method described above.

[0017] In a text data processing method provided by the present application, the plurality of text data is classified to obtain a category to which each text data belongs; for each category, a text similarity between each two text data in the category is calculated; a diversity evaluation result of each category is determined according to the text similarity between each two text data in the category; a category whose diversity evaluation result meets a preset enhancement condition is determined as a to-be-enhanced category, and data enhancement processing is performed on the text data in the to-be-enhanced category. As can be seen, the method classifies the text data, calculates the diversity evaluation result of each category respectively, determines the to-be-enhanced category, and performs data enhancement processing on the text data in the category; according to scientific analysis, the part of text data that needs data enhancement is obtained and processed, avoiding the problem of resource waste caused by blind global data expansion, and improving the generalization ability of the model.

[0018] It is to be understood that the embodiments described herein are merely exemplary of the application and that a person skilled in the art can make many variations and modifications without departing from the scope and spirit of the application. Also, for clarity and conciseness, the description that follows can omit descriptions of well-known functions and constructions. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. The above and other features and advantages of the present application will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flow chart of a text data processing method provided for Embodiment One of the present application;

[0021] Figure 2 A flow chart of a text data processing method provided for Embodiment Two of the present application;

[0022] Figure 3 A flow chart of a text data processing method provided for a specific example of Embodiment Two of the present application;

[0023] Figure 4 A structural block diagram of a text data processing device provided for Embodiment Three of the present application;

[0024] Figure 5 A structural block diagram of an electronic device provided for Embodiment Four of the present application. DETAILED DESCRIPTION

[0025] In order for those skilled in the art to better understand the technical solutions of the present application, the following describes exemplary embodiments of the present application in conjunction with the drawings, which include various details of the embodiments of the present application to help understanding, and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for clarity and conciseness, the following description omits descriptions of well-known functions and structures.

[0026] In the case of no conflict, each embodiment of the present application and each feature in the embodiments can be combined with each other.

[0027] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. "Coupled" or "connected" or similar terms are not restricted to physical or mechanical connections or associations, but can also include electrical connections, whether direct or indirect.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present application, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0030] The method for processing text data according to the embodiments of the present application can be executed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a physical server, a server cluster composed of multiple physical servers, or a distributed system, or a cloud server providing cloud computing services. The method can be implemented by a processor invoking a computer program stored in a memory.

[0031] In the related art, the diversity of training sample data is mainly increased by two ways of manual construction, screening, sample labeling and data enhancement, so as to improve the model training effect. However, for the above two ways, the former needs to consume a lot of manpower and material resources, and needs to be completed by professional personnel in the field; the latter is easy to introduce noise if not used properly, which affects the effect of the model. In order to solve the above problems, a method for processing text data is provided in the present application, which classifies the text data, respectively calculates the diversity evaluation result of each category, determines the category to be enhanced, and performs data enhancement processing on the text data in the category; according to scientific analysis, the part of text data that needs data enhancement is obtained and processed, avoiding the problem of resource waste caused by blind global data expansion, and improving the generalization ability of the model.

[0032] Embodiment one

[0033] Figure 1 A flowchart of a text data processing method provided for Embodiment One of the present application is shown in FIG. 1. Referring to FIG. 1, the method comprises the following steps. Figure 1

[0034] Step S110: classifying the plurality of text data to obtain a category to which each text data belongs.

[0035] The plurality of text data are used as training samples for model training and are obtained by manual pre-acquisition. The categories included in the classification are set by manual pre-setting. The classification of the plurality of text data is achieved by manual labeling of each text data. The category to which each text data belongs is obtained according to the classification result labeled by manual.

[0036] Step S120: calculating the text similarity between each two text data in each category, respectively.

[0037] The purpose of the text similarity analysis is to analyze and measure the distance between two texts. The text can be a simple identifier or a word frequency, such as a word, or an entire document containing a sentence and a text paragraph. Text similarity is mainly used in information retrieval, document duplication detection, and other fields to recommend similar content to users.

[0038] In a specific implementation, the calculation of the text similarity between the text data can be achieved by using the cosine similarity method. The cosine similarity, also known as the cosine similarity, is used to evaluate the similarity between two vectors by calculating the cosine value of the included angle between the two vectors. The cosine value is used to measure the difference between two individuals. When the cosine value is close to 1, the included angle tends to 0, indicating that the two vectors are more similar. When the cosine value is close to 0, the included angle tends to 90 degrees, indicating that the two vectors are less similar. The calculation of the text similarity between the text data is achieved by using the existing technology. The calculation method is selected by the person skilled in the art when implementing the step, and the present application does not limit the calculation method.

[0039] Step S130: determining a diversity evaluation result of each category according to the text similarity between each two text data in each category.

[0040] The diversity evaluation result is used to evaluate the data diversity of the text data. The diversity evaluation result is measured by using a pre-set index, such as the dispersion between the texts, the average value of the text similarity, etc. The parameters required in the above measurement index are further calculated based on the text similarity calculated in step S120. The index used to measure the diversity evaluation result is set by the person skilled in the art when implementing the step, and the present application does not limit the index.​

[0041] For example, in one specific example, first, for each similarity interval in a plurality of similarity intervals, the number of intervals in which the similarity value between each two text data in each category is located is counted, and the ratio between the number of intervals and the total number of similarity values between each two text data in the category is determined as the category interval proportion of the category corresponding to the similarity interval; wherein the plurality of similarity intervals are divided according to the value range of the text similarity; and the total number of similarity values between each two text data in the category is determined according to the total number of text data contained in the category; then, according to the category interval proportion of the category corresponding to each similarity interval, the diversity evaluation result of the category is determined.

[0042] Step S140: determining the category whose diversity evaluation result meets the preset enhancement condition as the to-be-enhanced category, and performing data enhancement processing on the text data in the to-be-enhanced category.

[0043] Wherein, the preset enhancement condition is a comparison relationship between the diversity evaluation result and the measurement index, and meeting the preset enhancement condition indicates that the data diversity of the classified text data is low, and data enhancement is needed, so it is determined as the to-be-enhanced category. The above-mentioned preset enhancement condition is set by the person skilled in the art when implementing this step according to the specific situation, which is not limited in this application.

[0044] Wherein, the data enhancement processing can be realized by synonym replacement, random insertion, random exchange, random deletion and the like; wherein, synonym replacement refers to randomly selecting a plurality of non-stop words in a sentence, and replacing each selected word with a randomly selected synonym; random insertion refers to randomly selecting a synonym and inserting it into any position in the sentence; random exchange refers to randomly selecting two words in the sentence and exchanging their positions; and random deletion refers to randomly deleting each word in the sentence with a certain probability. In one specific implementation, the text data enhancement is realized by EDA, AEDA and the like; wherein, EDA (Easy Data Augmentation) is a simple data enhancement technique for improving the performance of text classification tasks, which consists of four simple operations: synonym replacement, random insertion, random exchange and random deletion; AEDA (An Easy Data Augmentation) realizes data enhancement by randomly inserting punctuation marks in the original text.

[0045] To sum up, in the text data processing method provided in the embodiment, the plurality of text data is classified to obtain the category to which each text data belongs; the text similarity between each two text data in each category is calculated respectively for each category; the diversity evaluation result of each category is determined according to the text similarity between each two text data in each category; the category whose diversity evaluation result meets the preset enhancement condition is determined as a to-be-enhanced category, and data enhancement processing is performed on the text data in the to-be-enhanced category. As can be seen, the method classifies the text data, respectively calculates the diversity evaluation result of each category, determines the to-be-enhanced category, and performs data enhancement processing on the text data in the category. According to scientific analysis, the part of text data that needs data enhancement is obtained and processed, avoiding the problem of resource waste caused by blind global data expansion, and improving the generalization ability of the model.

[0046] Embodiment Two

[0047] Figure 2 A flowchart of a text data processing method provided in Embodiment Two of the present application is provided. Referring to Figure 2 , the method comprises:

[0048] Step S210: classifying the plurality of text data to obtain the category to which each text data belongs.

[0049] The plurality of text data is used as a training sample for model training and is obtained by manual pre-acquisition. The categories included in the classification are set by manual pre-setting. The classification of the plurality of text data is realized by manual labeling of each text data, and the category to which each text data belongs is obtained according to the classification result of manual labeling.

[0050] Step S220: calculating the sample similarity between each two text data in each category respectively.

[0051] In an optional implementation, before calculating the sample similarity between each two pieces of text data in a category, text representation processing is performed on the text data in the category, and the text data is vectorized to have the same dimension for each piece of text data, for subsequent text similarity calculation; wherein the text representation can be implemented using methods such as BERT (Bidirectional Encoder Representations from Transformer, a pre-trained language representation model), SimCSE (Simple Contrastive Learning of Sentence Embeddings, a simple contrastive learning of sentence embeddings), Word2vec (a group of related models for generating word vectors), and the like.

[0052] In a specific implementation, for the calculation of the text similarity between the text data, methods such as cosine similarity can be used. Cosine similarity, also known as cosine similarity, is used to evaluate the similarity between two vectors by calculating the cosine value of the included angle between the two vectors; the cosine value of the included angle between two vectors in a vector space is used as a measure of the difference between the two individuals, and the cosine value approaches 1 and the included angle approaches 0, indicating that the two vectors are more similar, and the cosine value approaches 0 and the included angle approaches 90 degrees, indicating that the two vectors are less similar. The calculation of the text similarity between the text data is implemented by the prior art, and the calculation method is selected by the person skilled in the art when implementing this step, which is not limited in this application.

[0053] Step S230: For any category, according to the value range of the text similarity in the category, at least two non-overlapping similarity intervals are divided, and the category interval proportion corresponding to each similarity interval of the category is calculated.

[0054] First, for any category, the total number of similarity values between each two pieces of text data in the category is determined according to the total number of text data contained in the category; in other words, for any category, the similarity between each two pieces of text data in the category is calculated, and based on the total number of text data contained in the category, the total number of all similarity results in the category is obtained, i.e. the total number of similarity values.

[0055] Second, at least two non-overlapping similarity intervals are divided according to the value range of the text similarity. According to the value range of all text similarity values in the category calculated in step S220, a threshold is selected between [0, 1] to divide the intervals; for example, α is selected as the threshold, and two similarity intervals [0, α) and [α, 1] are divided. The selection of the threshold and the division of the similarity intervals are determined by the person skilled in the art when implementing the method, which is not limited in this application.

[0056] Finally, for each similarity interval, the number of intervals in which the similarity value between each two text data in the category is located is counted, and the ratio between the interval number and the total number of values is determined as the category interval proportion of the category corresponding to the similarity interval; for example, the total number of similarity values is sum, two similarity intervals [0, a) and [a, 1] are divided, the interval number located in the interval [0, a) is sum1, and the interval number located in the interval [a, 1] is sum2, then the category interval proportion corresponding to the similarity interval [0, a) can be determined as sum1 / sum, and the category interval proportion corresponding to the similarity interval [a, 1] is sum2 / sum.

[0057] Step S240: For any category, the diversity evaluation result of the category is determined according to the category interval proportion of the category corresponding to each similarity interval.

[0058] The diversity evaluation result can be represented by a category dispersion index and / or a category similarity mean. The two ways of the category dispersion index and the category similarity mean can be used in combination or separately, and the present application does not limit this.

[0059] In a specific implementation, the diversity evaluation result is represented by the category dispersion index, the dispersion refers to the deviation of the same index distribution from a certain central index distribution, and the greater the dispersion, the more excellent the diversity of the text data; then this step specifically includes: for any category, the category dispersion index of the category is determined according to the category interval proportion of the category corresponding to each similarity interval; specifically, the dispersion index of the category is calculated by the following formula:

[0060]

[0061] wherein ω θ is the dispersion index, p θj is the category interval proportion of the jth similarity interval corresponding to the θth category, the number of similarity intervals is n, and m is the total number of categories.

[0062] In a specific implementation, the diversity evaluation result is represented by the category similarity mean, the smaller the similarity mean, the smaller the similarity between the text data, and the more excellent the diversity of the text data; then this step specifically includes: for any category, the category similarity mean of the category is determined according to the text similarity between each two text data in the category; specifically, the similarity mean of the category is calculated by the following formula:

[0063]

[0064] wherein a total number of values for representing the similarity value in the θth category; mean θ is the similarity mean, N θ is the number of text data contained in the θth category; sim(text_i, text_j) represents the text similarity between the text data text_i and the text data text_j.

[0065] Step S250: Determine the category whose diversity evaluation result meets the preset enhancement condition as the to-be-enhanced category, and perform data enhancement processing on the text data in the to-be-enhanced category.

[0066] In a specific implementation, the diversity evaluation result is represented by a category dispersion index, and the preset enhancement condition includes that the category dispersion index is less than a preset dispersion threshold; wherein the dispersion threshold is determined according to an overall dispersion index, and the overall dispersion index is obtained according to a category interval proportion of each category corresponding to a similarity interval and a category weight of each category; wherein the category weight of each category is determined according to a ratio between a category sample number of the category and a total sample number; specifically, the overall dispersion index is calculated by the following formula:

[0067]

[0068] wherein ω is the dispersion index, p ij is the category interval proportion of the ith category corresponding to the jth similarity interval, is a ratio of the number of text data of the ith category to the total number of text data, represents the category weight of the ith category, the number of similarity intervals is n, and m is the total number of categories.

[0069] In a specific implementation, the diversity evaluation result is represented by a category similarity mean, and the preset enhancement condition includes that the category similarity mean is greater than a preset mean threshold; wherein the mean threshold is determined according to an overall similarity mean, and the overall similarity mean is obtained according to the text similarity between each two text data in the plurality of categories; specifically, the overall similarity mean is calculated by the following formula:

[0070]

[0071] wherein,

[0072]

[0073] wherein mean is the overall similarity mean, N is the number of text data contained in the ith category; sim(text_i, text_j) represents the text similarity between the text data text_i and the text data text_j. a number of values for representing the similarity values in the i-th category; I is an indicator function, which is 1 if sample text_i and sample text_j belong to the same category, otherwise 0.

[0074] In summary, the preset enhancement condition includes: ω θ > ω, and / or mean θ < mean, and the category meeting the condition is determined as the to-be-enhanced category.

[0075] After determining the to-be-enhanced category according to the preset enhancement condition, data enhancement processing needs to be performed on the text data in the to-be-enhanced category, specifically including: obtaining a plurality of text data in the to-be-enhanced category whose similarity values are located in a first similarity interval; wherein the upper limit of the first similarity interval is less than a first similarity threshold, for example, the first similarity threshold is a1, and the first similarity interval is [0, a1); performing data enhancement processing on the plurality of text data located in the first similarity interval; wherein the data enhancement processing includes at least one of the following: synonym replacement, back translation processing, and random insertion of punctuation.

[0076] In another specific implementation, performing data enhancement processing on the text data in the to-be-enhanced category includes:

[0077] obtaining a plurality of text data in the to-be-enhanced category whose similarity values are located in a second similarity interval; wherein the lower limit of the second similarity interval is greater than a second similarity threshold; wherein the second similarity threshold is not less than the first similarity threshold; for example, the first similarity threshold is a1, the second similarity threshold is a2, a1≤a2, and the first similarity interval is [a2, 1].

[0078] performing data reduction processing on the plurality of text data located in the second similarity interval; wherein the data reduction processing includes: de-duplication processing on duplicate data, and / or the data reduction processing includes: taking a piece of text data located in the second similarity interval as reference text data, and if the similarity between the reference text data and non-reference text data in the second similarity interval is greater than a preset reference value, removing the reference text data; in other words, the text data that is completely duplicated in the second similarity interval and the text data with high similarity are processed by the reduction processing; wherein the preset reference value used to evaluate the similarity between the reference text data and the non-reference text data in the second similarity interval is set by those skilled in the art when implementing the method according to specific circumstances, which is not limited here.

[0079] In an optional implementation, for all the above steps, the plurality of text data are a plurality of text data with different lengths; and the category to which each text data belongs includes a plurality of sentiment tendency categories; and the text similarity between the text data is obtained by: converting each text data into a corresponding text vector, and representing the text similarity between the text data by the vector similarity between the text vectors; and after performing the data augmentation processing on the text data in the to-be-enhanced category, the method further includes: taking the plurality of text data after performing the data augmentation processing as training samples, and training the sentiment classification model by using the training samples. The sentiment classification model is used to predict the corresponding sentiment tendency category for input text data.

[0080] To sum up, in the method for processing text data provided in this embodiment, the plurality of text data are classified to obtain the category to which each text data belongs; the text similarity between each two text data in each category is calculated respectively for each category; for any category, at least two non-overlapping similarity intervals are divided according to the value range of the text similarity in the category, and the category interval proportion corresponding to each similarity interval of the category is calculated to determine the diversity evaluation result of the category; the category whose diversity evaluation result meets the preset enhancement condition is determined as a to-be-enhanced category, and the data augmentation processing is performed on the text data in the to-be-enhanced category. As can be seen, the method classifies the text data, respectively calculates the diversity evaluation result of each category, determines the to-be-enhanced category, and performs the data augmentation processing on the text data in the category. According to the scientific analysis, the part of text data that needs data augmentation is obtained and processed, the calculation speed is fast, the automation degree of the process is high, the problem of resource waste caused by blind global data expansion is avoided, and the generalization ability of the model is improved.

[0081] For the sake of convenience, a specific example is taken below to describe the specific implementation details of this embodiment in detail.

[0082] In recent years, the problem of data diversity has attracted more and more attention in the field of machine learning, and has become a major challenge faced by machine learning models in actual production. When the diversity distribution of data does not meet the coverage rate of the actual scene, the predicted data diversity distribution and the data diversity distribution used for training show obvious deviation, which will result in poor performance of the model.

[0083] Data diversity problem leads to poor performance of the model, which is difficult to solve by improving the generalization ability of the model in this case, because the current machine learning method is basically based on the premise of independent and identically distributed. Under a real distribution, the limited training data observed can be observed, and the accuracy of the model trained in prediction when encountering the same distribution but unobserved samples is reduced. For this case, the generalization ability of the model can be effectively improved by selecting appropriate algorithms, cross-validation, regularization, etc. However, the essence of the problem of data diversity is that the real distribution of data is very different from the distribution of the actual scene, so only improving the generalization ability cannot effectively improve the model effect.

[0084] The method for solving the above difficulties in the prior art is: manually constructing, screening and annotating samples, which has good effect, but requires a lot of manpower and material resources, and requires professionals in the field; the data enhancement method, which is simple and easy to operate, can solve the problem of manpower and material resources, but if not used properly, it can introduce noise and affect the effect of the model, and has the following disadvantages: the semantics of the augmented sentence is not easy to control and is easy to change, which affects the generalization effect of the model; cannot increase diversity specifically, which can easily lead to a global effect that cannot be optimized.

[0085] Figure 3 A flowchart of a text data processing method is provided for the specific example of embodiment two of the present application. Referring to Figure Three , the method comprises:

[0086] Step one: classification of original annotated data;

[0087] Input the original annotated data, assuming that there are N samples (i.e. text data in embodiment two), respectively text_1, text_2, …, text_N; there are m categories, respectively C1, C2, …, C m ; the number of texts contained in each category is N1, N2, …, N m . Wherein, N=N1+N2+…+N m .

[0088] Step two: similarity calculation;

[0089] Text representation can be realized by using BERT, SimCSE, Word2vec, etc. Similarity calculation can be realized by using cosine similarity, etc. For categories C1, C2, …, C m , the similarity of any two texts in the category is calculated.

[0090] Step three: data statistics;

[0091] For categories C1, C2, …, C m, the proportion of the similarity value of the text included in each category in the interval [0, a1), [a1, a2), [a2, a3), [a3, 1] is p 11 ,p 12 ,p 13 ,p 14 , p 21 ,p 22 ,p 23 ,p 24 , …, p m1 ,p m2 ,p m3 ,p m4 ; wherein the values of a1, a2, a3 are generally 0.5, 0.7, 0.9.

[0092] Step four: overall diversity calculation;

[0093] The overall diversity is used to measure the data diversity of the whole set of labeled data, and the overall diversity is jointly measured by ω and mean, and the calculation formulas are respectively:

[0094]

[0095]

[0096] wherein,

[0097]

[0098] wherein, ω is the overall dispersion index in example two, which is used to measure the dispersion between samples, the larger the value, the greater the dispersion between samples, indicating that the diversity of the sample is more excellent; mean is the overall similarity mean in example two, which represents the average value of the similarity between all samples, the smaller the value, the smaller the similarity between samples, indicating that the diversity of the sample is more excellent; is the ratio of the number of samples of category C i to the total number of labeled data samples, indicating the sample weight of category C i ; sim(text_i, text_j) represents the text similarity between sample text_i and sample text_j; I is an indicator function, which is 1 when sample text_i and sample text_j belong to the same category, otherwise it is 0.

[0099] Step five: inter-class diversity calculation;

[0100] The inter-class diversity is used to measure the data diversity of each category of labeled data, and the diversity of the θth category C θ is measured by

[0101] ω θand mean θ The common measure is calculated by the following formula:

[0102]

[0103]

[0104] wherein,

[0105] wherein, ω θ is the average of the category similarity in Example Two, which is used to measure the purity of the samples in the category, that is, the dispersion of the samples in the category, and indirectly measures the overall similarity between the samples, that is, the pros and cons of the diversity, the greater the value, the better the diversity of the samples in the category; mean θ is the average of the category similarity in Example Two. The overall diversity formula has one more weight than the inter-class diversity formula, because the number of samples in each category is different, so the weight of each category needs to be set to make the result less biased.

[0106] Step six: diversity judgment;

[0107] The judgment criterion for whether the diversity of the θth category C θ meets the requirements is:

[0108] ω θ > ω, and mean θ < mean

[0109] If ω θ and mean θ meet the judgment criterion, it means that the diversity of C θ in the original data is sufficient, and there is no need to enrich the inter-class diversity of this category; otherwise, it means that the diversity of C θ is not enough, and the inter-class diversity of this category needs to be enriched.

[0110] Step seven: enriching diversity;

[0111] For categories with insufficient diversity, the inter-class diversity is enriched by the following methods:

[0112] (1) Reduce the number of samples:

[0113] ① Remove the samples whose similarity values are distributed in the interval [α3, 1];

[0114] ② For samples whose similarity values are distributed in the interval [α3, 1], if the similarity between one sample and any other sample exceeds , remove the sample;

[0115] (2) Increase the number of samples:

[0116] For the samples whose similarity values are distributed in the intervals [0, a1), [a1, a2), [a2, a3), data augmentation is performed, and the more the sample number is increased for the samples with lower intervals of similarity values; wherein the data augmentation method can use synonym replacement, back translation, random insertion of punctuation, etc.

[0117] To sum up, in the text data processing method provided in the specific example, the diversity of the samples is detected and the generalization ability of the model is improved by statistically annotating the similarity values of the samples between classes in the data set, designing the evaluation method of overall diversity and inter-class diversity, the diversity judgment rule and the diversity enrichment method. Among them, for the evaluation method of diversity, a new sample diversity measurement method is provided, which can quickly and effectively calculate the overall and inter-class diversity of the samples, avoid blind model training, and increase the business cycle time; for the diversity judgment rule, the class with insufficient inter-class diversity can be evaluated and determined; for the diversity enrichment, a scientific method for enriching the inter-class diversity is provided, which improves the generalization ability of the model and solves the problem of blind global data augmentation, which wastes resources (time cost, machine cost, etc.) and even has a negative effect (easy to introduce noise, cause data pollution; the semantics of augmented data is not easy to control; cannot achieve global optimization); in addition, the method has the characteristics of fast calculation speed and high degree of automation.

[0118] The method provided by the specific examples above is mainly applied to the scene of emotion recognition. In the scene of emotion recognition, the sentiment polarity of the text needs to be accurately classified through model training, and the polarity of the emotion generally includes three categories of positive, negative and neutral (in the subdivided scene, the positive can be further divided into happy, love, etc., and the negative can be further divided into complaint, sadness, anger, etc.). In this scene, the characteristics of the text in each category are: the length of the text is various; the expression of the text in each category is various (i.e. the texts are not similar, but the semantics or labels can be the same; for example, "this fruit is delicious" and "today the weather is good, the sky is blue, the breeze is blowing, and going to the countryside is a good choice", which can be considered as positive emotion). In this scene, the statistical method based on word, sentence and other dimensions and the model verification method cannot objectively reflect the diversity of the data. The above method measures the diversity of the data from the perspective of the vector representation of the text. Before model training, a batch of text data is randomly selected in the scene, and the corresponding label is marked on the text through manual labeling to obtain the original labeled data set; the overall labeled data and the text data of each category are counted, including the total number of samples, the number of categories, and the number of samples in each category; for each category of the labeled data set, the similarity between any two texts in the category is calculated; for the similarity calculation value of the samples in each category, the proportion of the texts in each category whose similarity value is in different similarity value intervals is counted; according to the above statistical data and the diversity calculation formula, the overall diversity and inter-class diversity of the labeled data are calculated; through the diversity judgment rule, the category with insufficient inter-class diversity is determined; for the category with insufficient diversity, the number of samples whose similarity value is distributed in different intervals is reduced or increased to solve the problem of inter-class diversity; the corrected labeled data set is used for model training for text sentiment polarity recognition.

[0119] In summary, the present application is not only suitable for the sentiment classification and recognition scenario, but also suitable for other similar scenarios; not only suitable for the scenario with great difference in text length, but also suitable for the scenario with little difference in text length. For example, in the above-mentioned sentiment classification and recognition scenario, the characteristics of the text in each category are that the text length is different, and the expression of the text in each category is various (that is, the texts are not similar, but the semantics or labels can be the same. For example, "This fruit is delicious" and "Today the weather is good, the sky is clear, the breeze is blowing, and it is a good choice to go to the countryside." can be considered as positive sentiment.). As can be seen, in the case of great difference in text length and very extensive expression in each category, the present application is more superior. For example, "This fruit is delicious" and "Today the weather is good, the sky is clear, the breeze is blowing, and it is a good choice to go to the countryside." If the traditional statistical method based on word, sentence and other dimensions is used to judge the difference, they are almost completely irrelevant, but using the scheme in the present application can obtain a certain degree of relevance (because they belong to the same category and are also positive expressions in semantics, they should also have a certain degree of relevance.). Therefore, the present application can better illustrate the data diversity problem for similar scenarios.

[0120] It can be understood that the above-mentioned various method embodiments mentioned in the present application can be combined with each other to form combined embodiments without violating the principle logic. Limited by the length, the present application will not be described again. Those skilled in the art can understand that in the above-mentioned method of the specific implementation mode, the specific execution order of each step should be determined according to its function and possible internal logic.

[0121] Embodiment three

[0122] Figure 4 A structural block diagram of a text data processing device provided by the embodiment three of the present application is shown in FIG. 4. Referring to FIG. 4, Figure 4 The embodiment three of the present application provides a text data processing device 40, which comprises:

[0123] A text classification module 41, configured to classify a plurality of text data to obtain a category to which each text data belongs;

[0124] A similarity calculation module 42, configured to calculate a text similarity between each two text data in each category respectively;

[0125] A diversity evaluation module 43, configured to determine a diversity evaluation result of each category according to the text similarity between each two text data in each category;

[0126] The data augmentation module 44 is configured to determine a category in which the diversity evaluation result meets a preset augmentation condition as a to-be-augmented category, and perform data augmentation processing on text data in the to-be-augmented category.

[0127] Optionally, the diversity evaluation module 43 is specifically configured to:

[0128] For any category, the total number of similarity values between each two pieces of text data in the category is determined according to the total number of the text data contained in the category.

[0129] At least two non-overlapping similarity intervals are divided according to the value range of the text similarity.

[0130] For each similarity interval, the number of intervals in which the similarity value between each two pieces of text data in the category is located is counted, and the ratio between the number of intervals and the total number of values is determined as the category interval proportion of the category corresponding to the similarity interval.

[0131] The diversity evaluation result of the category is determined according to the category interval proportion of the category corresponding to each similarity interval.

[0132] Optionally, the diversity evaluation result is represented by a category dispersion index, and the diversity evaluation module 43 is specifically configured to:

[0133] For any category, the category dispersion index of the category is determined according to the category interval proportion of the category corresponding to each similarity interval.

[0134] The preset augmentation condition includes that the category dispersion index is less than a preset dispersion threshold, wherein the dispersion threshold is determined according to an overall dispersion index, and the overall dispersion index is obtained according to the category interval proportion of each category corresponding to the similarity interval and the category weight of each category, wherein the category weight of each category is determined according to the ratio between the category sample number of the category and the total sample number.

[0135] Optionally, the diversity evaluation result is represented by a category similarity mean, and the diversity evaluation module 43 is specifically configured to:

[0136] For any category, the category similarity mean of the category is determined according to the text similarity between each two pieces of text data in the category.

[0137] The preset augmentation condition includes that the category similarity mean is greater than a preset mean threshold, wherein the mean threshold is determined according to an overall similarity mean, and the overall similarity mean is obtained according to the text similarity between each two pieces of text data in the plurality of categories.

[0138] Optionally, the data augmentation module 44 is specifically configured to:

[0139] obtain a plurality of text data in the to-be-enhanced category with a similarity value in a first similarity interval; wherein an upper limit of the first similarity interval is less than the first similarity threshold;

[0140] perform data enhancement processing on the plurality of text data in the first similarity interval; wherein the data enhancement processing comprises at least one of the following: synonym replacement, back translation processing, and random insertion of punctuation.

[0141] Optionally, the data enhancement module 44 is specifically configured to:

[0142] obtain a plurality of text data in the to-be-enhanced category with a similarity value in a second similarity interval; wherein a lower limit of the second similarity interval is greater than a second similarity threshold; wherein the second similarity threshold is not less than the first similarity threshold;

[0143] perform data reduction processing on the plurality of text data in the second similarity interval;

[0144] wherein the data reduction processing comprises: de-duplication processing on duplicate data, and / or,

[0145] the data reduction processing comprises: taking a text data in the second similarity interval as reference text data, and if a similarity between the reference text data and non-reference text data in the second similarity interval is greater than a preset reference value, removing the reference text data.

[0146] Optionally, the plurality of text data is a plurality of text data with different lengths; and the category to which each text data belongs comprises a plurality of sentiment orientation categories.

[0147] Optionally, the similarity calculation module 42 specifically comprises:

[0148] convert each text data into a corresponding text vector, and represent text similarity between text data through vector similarity between text vectors.

[0149] Optionally, the data enhancement module 44 further comprises:

[0150] use the plurality of text data after performing the data enhancement processing as training samples, and train the sentiment classification model through the training samples.

[0151] The specific structure and working principle of each module can be referred to the description of the corresponding part of the method embodiments one and two, which will not be repeated here.

[0152] Embodiment four

[0153] Figure 5 A structural block diagram of an electronic device provided in embodiment four of the present application. Referring to Figure 5Embodiment four of the present application provides an electronic device 50, the electronic device 50 comprising:

[0154] at least one processor 501; a memory 502 connected with the at least one processor 501; and one or more I / O interfaces 503 connected between the at least one processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs executable by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to enable the at least one processor 501 to perform the above-mentioned text data processing method.

[0155] Embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned text data processing method. The computer readable storage medium can be a volatile or non-volatile computer readable storage medium.

[0156] Embodiments of the present application also provide a computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-mentioned text data processing method.

[0157] Those skilled in the art can understand that all or some steps in the above disclosed method, functions of modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable storage medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media).

[0158] As those skilled in the art will appreciate, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable program instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as those skilled in the art will appreciate, communication media typically embodies computer readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. As used herein, the term "exemplary" means serving as an example, instance, or illustration. Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations.

[0159] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0160] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0161] The computer program product described herein can be embodied in a specific manner by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), and the like.

[0162] The computer program product described herein can be embodied in a specific manner by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), and the like.

[0163] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, to cause a series of operational elements or steps to be performed on the computer to produce a computer implemented process; such that the instructions, which execute via one or more computer program product, implement a computer implemented process for performing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0164] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational elements or steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable data processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0165] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0166] Example embodiments have been disclosed herein and, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that features, characteristics or elements described with respect to a particular embodiment can be used, alone or in combination, with other embodiments unless specifically recited otherwise in the detailed description. Accordingly, it will be understood that various changes in form and details can be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A method of processing text data, characterized by, The method comprises: classifying a plurality of text data to obtain a category to which each text data belongs; for each category, calculating a text similarity between each two text data in the category; determining a diversity evaluation result of each category according to the text similarity between each two text data in the category by counting the distribution proportion of the text similarity value in a plurality of preset similarity intervals; determining a category that meets a preset enhancement condition as a to-be-enhanced category, and performing data enhancement processing on the text data in the to-be-enhanced category.

2. The method of claim 1, wherein, The method comprises: for each similarity interval in the plurality of similarity intervals, counting the number of similarity values between each two text data in the category that are located in the similarity interval, and determining a category interval proportion of the category corresponding to the similarity interval as a ratio between the number and a total number of similarity values between each two text data in the category; wherein the plurality of similarity intervals are divided according to the value range of the text similarity; and the total number of similarity values between each two text data in the category is determined according to the total number of text data included in the category; determining the diversity evaluation result of the category according to the category interval proportion of the category corresponding to each similarity interval.

3. The method of claim 2, wherein, The diversity evaluation result is represented by a category dispersion index and / or a category similarity mean.

4. The method of claim 3, wherein, In the case that the diversity evaluation result is represented by the category dispersion index, the method comprises: for any category, determining a category dispersion index of the category based on information entropy according to the category interval proportion of the category corresponding to each similarity interval; The preset enhancement condition comprises that the category dispersion index is less than a preset dispersion threshold; and determining a category that meets the preset enhancement condition as a to-be-enhanced category comprises determining a category whose category dispersion index is less than the preset dispersion threshold as the to-be-enhanced category; wherein the dispersion threshold is determined according to an overall dispersion index, and the overall dispersion index is obtained according to the category interval proportion of each category corresponding to a similarity interval and a category weight of each category; wherein the category weight of each category is determined according to a ratio between a category sample number of the category and a total sample number.

5. The method of claim 3, wherein, In the case that the diversity evaluation result is represented by the category similarity mean, the method comprises: for any category, determining a category similarity mean of the category according to the text similarity between each two text data in the category. The preset enhancement condition comprises: a category similarity mean value being greater than a preset mean value threshold; wherein the mean value threshold is determined according to an overall similarity mean value, and the overall similarity mean value is obtained according to text similarities between each two text data in each category in the plurality of categories.

6. The method of claim 3, wherein, The data enhancement processing performed on the text data in the category to be enhanced comprises: obtaining a plurality of text data in the category to be enhanced, similarity values of which are located in a first similarity interval; wherein an upper limit of the first similarity interval is less than a first similarity threshold; performing data enhancement processing on the plurality of text data located in the first similarity interval; wherein the data enhancement processing comprises at least one of the following: synonym replacement, back translation processing, and random insertion of punctuation.

7. The method of claim 6, wherein, The data enhancement processing performed on the text data in the category to be enhanced comprises: obtaining a plurality of text data in the category to be enhanced, similarity values of which are located in a second similarity interval; wherein a lower limit of the second similarity interval is greater than a second similarity threshold; wherein the second similarity threshold is not less than the first similarity threshold; performing data reduction processing on the plurality of text data located in the second similarity interval; wherein the data reduction processing comprises: de-duplication processing of duplicate data, and / or, the data reduction processing comprises: taking a piece of text data located in the second similarity interval as reference text data, and if a similarity between the reference text data and non-reference text data in the second similarity interval is greater than a preset reference value, removing the reference text data.

8. The method of claim 1, wherein, The plurality of text data are a plurality of text data with different lengths; and the category to which each text data belongs comprises a plurality of sentiment orientation categories; The text similarity between the text data is obtained by: converting each piece of text data into a corresponding text vector, and representing the text similarity between the text data by a vector similarity between the text vectors; and after the data enhancement processing is performed on the text data in the category to be enhanced, the method further comprises: taking the plurality of text data after the data enhancement processing is performed as training samples, and training a sentiment classification model by using the training samples; wherein the sentiment classification model is used to predict a corresponding sentiment orientation category for input text data.

9. An apparatus for processing text data, characterized by The method comprises: a text classification module, configured to classify a plurality of text data to obtain a category to which each piece of text data belongs; a similarity calculation module, configured to calculate a text similarity between each two pieces of text data in each category; a diversity evaluation module, configured to determine a diversity evaluation result of each category according to a distribution proportion of a text similarity value in each category in a plurality of preset similarity intervals; a data enhancement module, configured to determine a category whose diversity evaluation result meets a preset enhancement condition as a category to be enhanced, and perform data enhancement processing on text data in the category to be enhanced.

10. An electronic device, comprising: The method comprises: at least one processor; a memory connected in communication with the at least one processor; and one or more I / O interfaces connected between the at least one processor and the memory; wherein the memory stores one or more computer programs executable by the at least one processor, one or more of the computer programs are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the method of any one of claims 1-8. The computer program, when executed by the processor, implements the method of any one of claims 1-8.

Images