Data enhancement method and device

Abstract

The present disclosure provides a data enhancement method and apparatus. The data enhancement method includes: segmenting multiple training sentences with category identifiers, and identifying the category identifier corresponding to each word in the training sentence; performing a chi-square test on the words to obtain a characteristic value for each word in relation to the category identifier; performing noise processing on the training sentences based on the characteristic value and the words; and merging the noise-processed training sentences with the original training sentences to form a new training sentence set. The data enhancement method provided by the present disclosure can improve the accuracy of newly generated training data while compensating for insufficient model training data.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to a data enhancement method and device. Background Art

[0002] The effectiveness of deep learning neural network model training depends heavily on the training data. Large amounts of data with diverse features yield better training results. However, in reality, large amounts of data are often difficult to obtain, requiring extensive manual annotation, which consumes both financial and human resources. Data augmentation is an effective means of addressing the lack of training data. Data augmentation involves transforming limited training data to generate new data. Data augmentation can generate richer training data, thereby improving model performance.

[0003] Customer service robots are a new application area for data augmentation. With the increasing volume of inquiries, assessing the emotions expressed by customer input can improve the accuracy of customer service robots' responses. Therefore, a large amount of training data is required to train customer service robots to recognize the emotions expressed in sentences. To compensate for this lack of training data, data augmentation is performed on some training sentences. Noise processing is a common data augmentation technique, which involves randomly replacing elements in the original training data to generate new training data. In the field of semantic recognition, data augmentation often involves randomly replacing individual Chinese characters in a text with noise, following English conventions, to generate new text training data. However, in Chinese, individual characters often cannot accurately express emotional tendencies. Randomly replacing characters with noise can disrupt the original meaning of words constructed based on these characters. In sentiment classification, certain words are particularly sensitive to emotion. For example, words like "thank you" can clearly convey anger or happiness. Noising these words can affect the analysis of text sentiment.

[0004] Therefore, a training data enhancement method suitable for customer service robots to recognize semantic emotions is needed.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a data enhancement method and a data enhancement device, which are used to overcome, at least to a certain extent, the problem of using random text replacement in the data enhancement process to destroy the semantics of the original training data due to the limitations and defects of related technologies.

[0007] According to a first aspect of an embodiment of the present disclosure, a data enhancement method is provided, comprising: performing word segmentation on a plurality of training sentences with category identifiers, and identifying the category identifier corresponding to each vocabulary word; performing a chi-square test on the vocabulary words to obtain a feature value of each vocabulary word corresponding to the category identifier; performing noise processing on the training sentences based on the feature value and the vocabulary words; and merging the noise-processed training sentences with the original training sentences into a new training sentence set.

[0008] In an exemplary embodiment of the present disclosure, performing noise processing on the training sentence according to the feature value and the vocabulary includes:

[0009] Calculate the total frequency of each word and the number of distinct types of the following words;

[0010] Setting the replacement probability P1 of a word whose characteristic value is greater than a first preset value or whose word frequency is less than a second preset value to 0;

[0011] Setting the probability P2 of the vocabulary whose word frequency is greater than the third preset value to be replaced as the first parameter λ;

[0012] The probability of replacing a word whose word frequency is between the second preset value and the third preset value and whose characteristic value is less than or equal to the first preset value is set to P3=λ*distinct / total;

[0013] The words in the training sentence are replaced according to their corresponding replacement probabilities.

[0014] In an exemplary embodiment of the present disclosure, replacing the vocabulary in the training sentence according to the corresponding replacement probabilities includes:

[0015] The probability of each word appearing in all the training sentences is used as the probability of the word participating in the replacement;

[0016] Selecting a replacement word according to the probability of participating in the replacement, and selecting a replaced word from a plurality of identical words according to the probability of being replaced;

[0017] The replaced word is replaced with the replacement word.

[0018] In an exemplary embodiment of the present disclosure, the word segmentation of the plurality of training sentences with category identifiers includes:

[0019] Determining the number of training sentences corresponding to each category identifier;

[0020] Segment the training sentences corresponding to the category identifiers whose quantity meets the preset conditions.

[0021] In an exemplary embodiment of the present disclosure, segmenting the training sentences corresponding to the category identifiers whose number meets the preset conditions includes:

[0022] Determine the proportion of n numbers corresponding to n category identifiers according to the ratio of the number of training sentences corresponding to the category identifier to the total number of training sentences;

[0023] When the standard deviation of the n quantity proportions is greater than a fourth preset value, the category identifier corresponding to the quantity proportion greater than the fifth preset value and less than the sixth preset value is used as the category identifier to be enhanced;

[0024] When the standard deviation of the n number proportions is less than or equal to the fourth preset value, the category identifier whose number of corresponding training sentences is less than the seventh preset value is used as the category identifier to be enhanced;

[0025] Perform word segmentation on the training sentence corresponding to the category identifier to be enhanced.

[0026] In an exemplary embodiment of the present disclosure, the category identifier includes an emotion identifier.

[0027] In an exemplary embodiment of the present disclosure, it further includes:

[0028] While marking the category identifier corresponding to the training sentence for each vocabulary word, calculate and record the word vector of each vocabulary word:

[0029] The preset model is trained based on the word vectors of the training sentences in the new training sentence set.

[0030] According to a second aspect of an embodiment of the present disclosure, a data enhancement device is provided, including:

[0031] A word segmentation module is configured to segment a plurality of training sentences with category identifiers and to identify the category identifier corresponding to each word in the training sentence;

[0032] A chi-square test module, configured to perform a chi-square test on the vocabulary to obtain a feature value of each vocabulary corresponding to its category identifier;

[0033] a noise processing module configured to perform noise processing on the training sentence according to the feature value and the vocabulary;

[0034] The collection module is configured to merge the noise-processed training sentences with the original training sentences into a new training sentence set.

[0035] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the above methods based on instructions stored in the memory.

[0036] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the data enhancement method as described in any one of the above items is implemented.

[0037] The data enhancement method provided by the embodiment of the present disclosure processes the word segmentation results by using the chi-square test, determines the restrictive words that are highly correlated with the category identification, and avoids replacing the restrictive words when subsequently expanding the training samples. It can effectively retain the restrictive words that have a great impact on the category identification, and avoid replacing the key words during the data enhancement process, which causes the newly generated training data to be unable to correctly express the semantics of the original training data.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0040] Figure 1 is a flow chart of a data enhancement method in an exemplary embodiment of the present disclosure.

[0041] Figure 2 It is a sub-flowchart of step S3.

[0042] Figure 3 It is a sub-flowchart of step S35.

[0043] Figure 4 It is a flowchart for selecting training sentences.

[0044] Figure 5 This is a flowchart of selecting category identification for data enhancement in an embodiment of the present disclosure.

[0045] Figure 6 It is a block diagram of a data enhancement device in an exemplary embodiment of the present disclosure.

[0046] Figure 7 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure.

[0047] Figure 8 is a schematic diagram of a computer-readable storage medium in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0049] The accompanying drawings are merely schematic illustrations of the present disclosure. Identical reference numerals in the drawings denote identical or similar components, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0050] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0051] Figure 1 The flowchart of the data enhancement method in the exemplary embodiment of the present disclosure is schematically shown. Figure 1 , the data enhancement method 100 may include:

[0052] Step S1, segmenting a plurality of training sentences with category identifiers, and marking each word with a category identifier corresponding to the training sentence;

[0053] Step S2, performing a chi-square test on the vocabulary to obtain a characteristic value of each vocabulary corresponding to its corresponding category identification;

[0054] Step S3, performing noise processing on the training sentence according to the feature value and the vocabulary;

[0055] Step S4: Merge the noise-processed training sentences and the original training sentences into a new training sentence set.

[0056] The data enhancement method provided by the embodiment of the present disclosure processes the word segmentation results by using the chi-square test, determines the restrictive words that are highly correlated with the category identification, and avoids replacing the restrictive words when subsequently expanding the training samples. It can effectively retain the restrictive words that have a great impact on the category identification, and avoid replacing the key words during the data enhancement process, which causes the newly generated training data to be unable to correctly express the semantics of the original training data.

[0057] Below, each step of the data enhancement method 100 is described in detail.

[0058] In step S1, a plurality of training sentences with category identifiers are segmented, and each word is identified with a category identifier corresponding to the training sentence.

[0059] In the embodiment of the present disclosure, the category identifier includes an emotion identifier, such as happy and grateful, angry, lost, anxious, confused, worried, and other.

[0060] Table 1 is an example of the correspondence between training sentences and category identifiers.

[0061] Table 1

[0062] Training statements Emotional markers Okay, thanks. happy Why not refund the shipping fee? anger I'm drunk too. sad It's been a month, why hasn't it been shipped yet? anxiety Why didn't I receive a refund for my clothes? lost Is this a scammer who receives money but doesn't ship the goods? fear OK, I'll go check it out. other

[0063] The general word segmentation method can be used to segment the training sentences. For example, the text "If there is still a problem with the exchange, I will complain" with the category identifier "anger" will be segmented into "If / exchange / still / has / a / problem / I / will / complain / "; the text "Why is there no news about my return yet?" with the category identifier "anxiety" will be segmented into "Why is there / no / news / about / my / return / yet / ?"

[0064] After tokenizing the training sentences, each token can be labeled with the category identifier corresponding to the training sentence. For example, the category identifier for "complaint" could be "anger." Generally speaking, some words can appear in training sentences with different category identifiers. These can be recorded separately without further processing. During the token segmentation process, each word can also be labeled with its category, such as noun or verb, based on a pre-set vocabulary.

[0065] In step S2, a chi-square test is performed on the vocabulary to obtain a feature value of each vocabulary corresponding to its corresponding category identification.

[0066] The chi-square test is a hypothesis test method for counting narcissism, which can compare the correlation between two or more sample rates (composition ratios) and two categorical variables. By performing a chi-square test on vocabulary, we can determine the degree of association between each vocabulary and different category identifiers, that is, the size of the eigenvalue. The larger the eigenvalue, the higher the degree of association between the vocabulary and the vocabulary of that category. For example, the sentiment label of texts containing words such as "cheating people," "fake goods," and "complaint" is often anger, while words such as "refund" and "not arrived" are likely to be anxiety. For a vocabulary that appears in training sentences with different category identifiers, the eigenvalue of the vocabulary for each category identifier can be calculated separately, and the largest eigenvalue is used as the eigenvalue of the vocabulary.

[0067] In step S3, noise processing is performed on the training sentence according to the feature value and the vocabulary.

[0068] Figure 2 It is a sub-flowchart of step S3.

[0069] refer to Figure 2 In one embodiment, step S3 may include:

[0070] Step S31, calculate the total frequency of each word and the number of distinct types of the following words;

[0071] Step S32, setting the replacement probability P1 of a word whose characteristic value is greater than a first preset value or whose word frequency is less than a second preset value to 0;

[0072] Step S33, setting the replacement probability P2 of a word with a word frequency greater than a third preset value to a first parameter λ;

[0073] Step S34, setting the replacement probability of a word whose frequency is between the second preset value and the third preset value and whose characteristic value is less than or equal to the first preset value to P3 = λ*distinct / total;

[0074] Step S35: Replace the words in the training sentence according to their corresponding replacement probabilities.

[0075] The word frequency is the number of times a word appears in the word segmentation results of all training sentences, and the number of types of suffixes is the number of all types of suffixes of the word in all training sentences.

[0076] Among them, the successor of a word is the word immediately following it in the training sentences. When the frequency of a word is very high and the number of types of its successors is large, it can be considered that the characteristic of this word is not strong and it can be replaced with a relatively high probability without having a great impact on the meaning expressed by the training sentences. For example, the frequency of "not" and the number of types of its successors are 10,624 and 744 respectively, while the frequency of "want" and the number of types of its successors are 2,561 and 564 respectively. The tendency expressed by "not" is stronger and it should be replaced with a small probability, while "want" is more common and appears in many scenarios. Even if the replacement probability is large, it will not affect the semantic expression, so the replacement probability can be set to be relatively small.

[0077] Words with eigenvalues greater than the first preset value are words with a very high degree of association with a certain type identifier. Being replaced will affect the expression of the type identifier in the semantics. Therefore, in the embodiments of the present disclosure, the replacement probability of such words is set to 0, that is, they do not participate in replacement.

[0078] If words with a frequency less than the second preset value are replaced, the performance characteristics of such words may disappear. To avoid this situation, the replacement probability of such words is set to 0, that is, they do not participate in replacement.

[0079] After words with a frequency greater than the third preset value are processed by noise, their performance characteristics are less affected. Therefore, a relatively large replacement probability can be directly set for such words that appear frequently. It should be noted that the first parameter λ can be obtained as the reference probability, and the probability parameter can be set according to other situations, and the product of the reference probability and the probability parameter is used as the replacement probability. In the embodiments of the present disclosure, the replacement probability of words with a frequency greater than the third preset value can be set to the reference probability, that is, the probability parameter is set to 1. The above reference probability can be set by those skilled in the art themselves, and the present disclosure is not limited thereto.

[0080] For other words with small eigenvalues and medium frequencies, that is, other words except the above-mentioned words, the ratio of the number of types of successors to the frequency can be used as the probability parameter of this word, that is, the replacement probability of this word is set to P3 = λ * distinct / total.

[0081] After setting the replacement probability, each word can be replaced according to the replacement probability.

[0082] Figure 3 It is a sub-flowchart of step S35.

[0083] Reference Figure 3 , in one embodiment, step S35 may include:

[0084] Step S351, taking the occurrence probability of each word in all training sentences as the participation replacement probability of this word;

[0085] Step S352, selecting a replacement word according to the probability of participating in the replacement, and selecting a replaced word from multiple identical words according to the probability of being replaced;

[0086] Step S353: Use the replacement word to replace the replaced word.

[0087] If the probability of word A being replaced is 0.4, the word A to be replaced is determined based on the product of the word frequency of word A and the probability of being replaced. That is, if word A appears 100 times, 40 words A will be replaced.

[0088] After selecting the word to be replaced, you can choose a replacement word based on its probability of participation. For example, if the frequency of word B is x and the total number of words is y, then the probability of participation in the replacement of word B, P1, is P1 = x / y. If P1 = 0.2, then of the 40 words A to be replaced, 40 * 0.2 = 8 words A can be selected and replaced by word B.

[0089] After the replacement, words with low frequency and large eigenvalues ​​are not replaced, which ensures that the semantics of the training sentences will not be greatly affected.

[0090] In step S4, the noise-processed training sentences and the original training sentences are merged into a new training sentence set.

[0091] In the embodiment of the present disclosure, data enhancement is not performed on the training sentences of all category identifiers, so the new training sentence set may include the noise-processed training sentences and the original training sentences corresponding to only some category identifiers.

[0092] The embodiment of the present disclosure may further include a step of selecting a category identifier for data enhancement.

[0093] Figure 4 It is a flowchart for selecting training sentences.

[0094] refer to Figure 4 , step S1 may include:

[0095] Step S11, determining the number of training sentences corresponding to each category identifier;

[0096] Step S12: segmenting the training sentences corresponding to the category identifiers whose number meets the preset conditions.

[0097] Table 2 shows an example of the number of training sentences corresponding to various class identifiers.

[0098] Table 2

[0099] Category Identification quantity Proportion (%) All 51816 100 other 35213 67.96 anxiety 10009 19.32 anger 3913 7.55 happy 1246 2.40 lost 1240 2.39 sad 127 0.25 fear 68 0.13

[0100] In practical applications, the problem of imbalanced data among various categories of the dataset shown in Table 2 often occurs. Therefore, we only need to select the imbalanced category identifiers for data augmentation. There is no need to augment the category identifiers with too small a probability of occurrence (such as sad and fear), nor is there any need to augment the category identifiers with sufficient training data (such as other), which can save time.

[0101] Figure 5 This is a flowchart of selecting category identification for data enhancement in an embodiment of the present disclosure.

[0102] refer to Figure 5 In one embodiment, step S12 may include:

[0103] Step S121, determining n quantity proportions corresponding to n category identifiers according to the ratio of the number of training sentences corresponding to the category identifiers to the total number of training sentences;

[0104] Step S122: When the standard deviation of the n quantity proportions is greater than a fourth preset value, the category identifier corresponding to the quantity proportion greater than a fifth preset value and less than a sixth preset value is used as the category identifier to be enhanced;

[0105] Step S123: When the standard deviation of the n number proportions is less than or equal to the fourth preset value, the category identifier whose number of corresponding training sentences is less than the seventh preset value is used as the category identifier to be enhanced;

[0106] Step S124: performing word segmentation on the training sentence corresponding to the category identifier to be enhanced.

[0107] The proportion of each type of identification can be found in Table 2.

[0108] Whether the type distribution of these training sentences is balanced can be evaluated based on the proportion of the number and the standard deviation of each proportion of the number. When the standard deviation is greater than the fourth preset value, it can be considered that the type distribution of the training sentences is uneven. At this time, the type identifiers with too small a proportion (greater than the fifth preset value) can be eliminated first to improve efficiency, and then the type identifiers with a proportion that is not large enough (less than the sixth preset value) can be enhanced. Data enhancement. If the standard deviation is not greater than the fourth preset value, it can be considered that the type distribution of the training sentences is balanced. At this time, the type identifiers to be enhanced can be selected based solely on the number of training sentences, that is, data enhancement can be performed on the type identifiers whose number of training sentences does not meet the training requirements (seventh preset value).

[0109] The first to seventh preset values ​​can be set by those skilled in the art, and the present disclosure is not limited thereto.

[0110] After forming a new training sentence set, the model can be trained using the new training sentence set. It is worth mentioning that since the vocabulary of the training sentences needs to be expressed numerically, the word vector of each word can be determined at the same time as the word segmentation in step S1, and the word vector can be added to the new training sentence set as the data representation of the vocabulary.

[0111] In this disclosed embodiment, a CNN network used in image classification is applied to the text domain to extract deep semantic expressions of user emotions. The trained model primarily uses a logistic regression model corresponding to category identifiers. The model's sentence recognition results are scores for various category identifiers, with the size of the score representing the likelihood that the sentence belongs to a category identifier. The category identifier with the highest score is selected as the predicted category.

[0112] Tables 3 and 4 present comparative results on the validation and test datasets, respectively, based on JD.com customer service review data. The parameter λ was set to 0.35. Four different training data types were tested for each dataset: a baseline experiment (in which the input data was unprocessed); an experiment based on char segmentation; an experiment based on subword segmentation; and an experiment using subword segmentation and qualifiers selected using chi-square analysis. The F1 score was the evaluation metric used in the experiments. The table shows the F1 scores for the four different sentiment categories and the overall F1 score. As can be seen from the table, the overall F1 score for the char segmentation approach decreased by 0.004 on the validation dataset compared to the baseline, but improved by 0.012 on the test dataset. Subword segmentation decreased by 0.006 on the test dataset. The approach based on subword segmentation and qualifiers showed the greatest improvement in overall F1, increasing by 0.005 and 0.013 on the validation and test datasets, respectively, compared to the baseline. It can be seen that in sentiment classification, the noise processing method of subword segmentation and adding qualifiers to the original training data has the greatest benefit and the best classification effect compared with the other two methods.

[0113] Table 3. Comparison results of validation dataset experiments

[0114] Treatment method anxiety lost anger happy all baseline 0.909 0.590 0.829 0.860 0.859 char segmentation + noise 0.905 0.591 0.819 0.854 0.855 subword segmentation + noise 0.909 0.612 0.827 0.850 0.859 subword + determiner + noise 0.912 0.640 0.831 0.868 0.864

[0115] Table 4. Comparison results of test dataset experiments

[0116] Treatment method anxiety lost anger happy all baseline 0.773 0.482 0.700 0.800 0.695 char segmentation + noise 0.786 0.522 0.674 0.615 0.707 subword segmentation + noise 0.770 0.479 0.681 0.500 0.689 subword + determiner + noise 0.779 0.561 0.680 0.533 0.708

[0117] The disclosed embodiment performs word segmentation-based data enhancement processing on data with insufficient data and unbalanced data categories, and processes qualifiers and word segmentations of different word frequencies separately, so as to make the quality of the data after noise processing as high as possible, and the results of data enhancement do not change the original semantics as much as possible, thereby ensuring that the trained neural network training model can increase the recognition accuracy when sufficient training data is obtained.

[0118] Corresponding to the above method embodiments, the present disclosure also provides a data enhancement device, which can be used to execute the above method embodiments.

[0119] Figure 6 The following schematically shows a block diagram of a data enhancement device in an exemplary embodiment of the present disclosure.

[0120] refer to Figure 6 , the data enhancement device 600 may include:

[0121] The word segmentation module 602 is configured to segment a plurality of training sentences with category identifiers and identify the category identifier corresponding to each word in the training sentence;

[0122] A chi-square test module 604 is configured to perform a chi-square test on the vocabulary to obtain a feature value of each vocabulary corresponding to its category identifier;

[0123] a noise processing module 606 configured to perform noise processing on the training sentence based on the feature value and the vocabulary;

[0124] The combining module 608 is configured to combine the noise-processed training sentences with the original training sentences into a new training sentence set.

[0125] In an exemplary embodiment of the present disclosure, the noise processing module 606 is configured to:

[0126] Calculate the total frequency of each word and the number of distinct types of the following words;

[0127] Setting the replacement probability P1 of a word whose characteristic value is greater than a first preset value or whose word frequency is less than a second preset value to 0;

[0128] Setting the probability P2 of the vocabulary whose word frequency is greater than the third preset value to be replaced as the first parameter λ;

[0129] The probability of replacing a word whose word frequency is between the second preset value and the third preset value and whose characteristic value is less than or equal to the first preset value is set to P3=λ*distinct / total;

[0130] The words in the training sentence are replaced according to their corresponding replacement probabilities.

[0131] In an exemplary embodiment of the present disclosure, the replacing of the vocabulary in the training sentence according to the corresponding replacement probability in the noise processing module 606 includes:

[0132] The probability of each word appearing in all the training sentences is used as the probability of the word participating in the replacement;

[0133] Selecting a replacement word according to the probability of participating in the replacement, and selecting a replaced word from a plurality of identical words according to the probability of being replaced;

[0134] The replaced word is replaced with the replacement word.

[0135] In an exemplary embodiment of the present disclosure, the word segmentation module 602 is configured as follows:

[0136] Determining the number of training sentences corresponding to each category identifier;

[0137] Segment the training sentences corresponding to the category identifiers whose quantity meets the preset conditions.

[0138] In an exemplary embodiment of the present disclosure, the word segmentation module 602 performs word segmentation on the training sentences corresponding to the category identifiers whose number meets the preset conditions, including:

[0139] Determine the proportion of n numbers corresponding to n category identifiers according to the ratio of the number of training sentences corresponding to the category identifier to the total number of training sentences;

[0140] When the standard deviation of the n quantity proportions is greater than a fourth preset value, the category identifier corresponding to the quantity proportion greater than the fifth preset value and less than the sixth preset value is used as the category identifier to be enhanced;

[0141] When the standard deviation of the n number proportions is less than or equal to the fourth preset value, the category identifier whose number of corresponding training sentences is less than the seventh preset value is used as the category identifier to be enhanced;

[0142] Perform word segmentation on the training sentence corresponding to the category identifier to be enhanced.

[0143] In an exemplary embodiment of the present disclosure, the category identifier includes an emotion identifier.

[0144] In an exemplary embodiment of the present disclosure, a word vector recording module is further included, which is configured to:

[0145] While marking the category identifier corresponding to the training sentence for each vocabulary word, calculate and record the word vector of each vocabulary word:

[0146] The preset model is trained based on the word vectors of the training sentences in the new training sentence set.

[0147] Since the functions of the apparatus 600 have been described in detail in the corresponding method embodiments, they will not be described in detail herein.

[0148] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0149] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0150] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0151] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present invention will be described. Figure 7 The electronic device 700 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0152] like Figure 7 As shown, electronic device 700 is implemented as a general-purpose computing device. Components of electronic device 700 may include, but are not limited to, the aforementioned at least one processing unit 710, the aforementioned at least one storage unit 720, and a bus 730 connecting various system components (including storage unit 720 and processing unit 710).

[0153] The storage unit stores program codes that can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 may perform the following steps: Figure 1The steps S1 shown in the figure are as follows: performing word segmentation on a plurality of training sentences with category identifiers, and identifying the category identifier corresponding to each vocabulary word; step S2: performing a chi-square test on the vocabulary words to obtain the characteristic value of each vocabulary word corresponding to the category identifier; step S3: performing noise processing on the training sentences according to the characteristic value and the vocabulary words; step S4: merging the noise-processed training sentences with the original training sentences into a new training sentence set.

[0154] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202 , and may further include a read-only memory unit (ROM) 7203 .

[0155] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0156] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0157] The electronic device 700 can also communicate with one or more external devices 800 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 700, and / or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 750. Furthermore, the electronic device 700 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 700, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0159] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0160] refer to Figure 8 , a program product 900 for implementing the above-described method according to an embodiment of the present invention is described. The program product 900 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0162] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0163] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0164] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0166] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A data enhancement method, characterized in that: include: Segmenting a plurality of training sentences with category identifiers, and identifying each word with a category identifier corresponding to the training sentence; Performing a chi-square test on the vocabulary to obtain a feature value of each vocabulary corresponding to the category identification; performing noise processing on the training sentence according to the feature value and the vocabulary; Merge the noise-processed training sentences and the original training sentences into a new training sentence set; The performing noise processing on the training sentence according to the feature value and the vocabulary includes: Calculate the total frequency of each word and the number of distinct types of the following words; Setting the replacement probability P1 of a word whose characteristic value is greater than a first preset value or whose word frequency is less than a second preset value to 0; Setting the probability P2 of the vocabulary whose word frequency is greater than the third preset value to be replaced as the first parameter λ; The probability of replacing a word whose word frequency is between the second preset value and the third preset value and whose characteristic value is less than or equal to the first preset value is set to P3=λ*distinct / total; The words in the training sentence are replaced according to their corresponding replacement probabilities.

2. The data enhancement method according to claim 1, wherein: The replacing the words in the training sentence according to the corresponding replacement probabilities includes: The probability of each word appearing in all the training sentences is used as the probability of the word participating in the replacement; Selecting a replacement word according to the probability of participating in the replacement, and selecting a replaced word from a plurality of identical words according to the probability of being replaced; The replaced word is replaced with the replacement word.

3. The data enhancement method according to claim 1, wherein: The word segmentation of the plurality of training sentences with category identifiers includes: Determining the number of training sentences corresponding to each category identifier; Segment the training sentences corresponding to the category identifiers whose quantity meets the preset conditions.

4. The data enhancement method according to claim 3, wherein: The word segmentation of the training sentences corresponding to the category identifiers whose number meets the preset conditions includes: Determine the proportion of n numbers corresponding to n category identifiers according to the ratio of the number of training sentences corresponding to the category identifier to the total number of training sentences; When the standard deviation of the n quantity proportions is greater than a fourth preset value, the category identifier corresponding to the quantity proportion greater than the fifth preset value and less than the sixth preset value is used as the category identifier to be enhanced; When the standard deviation of the n number proportions is less than or equal to the fourth preset value, the category identifier whose number of corresponding training sentences is less than the seventh preset value is used as the category identifier to be enhanced; Perform word segmentation on the training sentence corresponding to the category identifier to be enhanced.

5. The data enhancement method according to claim 1, wherein: The category identifier includes an emotion identifier.

6. The data enhancement method according to claim 1, wherein: Also includes: While marking the category identifier corresponding to the training sentence for each vocabulary word, calculate and record the word vector of each vocabulary word: The preset model is trained based on the word vectors of the training sentences in the new training sentence set.

7. A data enhancement device, characterized in that: include: A word segmentation module is configured to segment a plurality of training sentences with category identifiers and to identify the category identifier corresponding to each word in the training sentence; A chi-square test module, configured to perform a chi-square test on the vocabulary to obtain a feature value of each vocabulary corresponding to its category identifier; a noise processing module configured to perform noise processing on the training sentence according to the feature value and the vocabulary; A collection module is configured to combine the noise-processed training sentences with the original training sentences into a new training sentence set; The noise processing module is configured to: calculate the word frequency total and the number of types of the following words distinct of each of the words; set the replacement probability P1 of the words whose characteristic value is greater than the first preset value or the word frequency is less than the second preset value to 0; set the replacement probability P2 of the words whose frequency is greater than the third preset value to the first parameter λ; set the replacement probability of the words whose frequency is between the second preset value and the third preset value and whose characteristic value is less than or equal to the first preset value to P3 = λ*distinct / total; and replace the words in the training sentence according to their corresponding replacement probabilities.

8. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the data enhancement method according to any one of claims 1 to 6 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the data enhancement method according to any one of claims 1 to 6 is implemented.

Images