A data enhancement method, device, equipment and medium
By obtaining the words to be replaced from the original text in NLP and determining their synonyms based on the test set vocabulary list for replacement, the problem of semantic changes and lack of targeted synonyms in the prior art is solved, and the data enhancement quality of the emotion classification model is improved and accuracy.
Patent Information
- Application Number
- CN202011035411.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-27
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2040-09-27
AI Technical Summary
The existing NLP data augmentation technology has problems such as semantic changes and lack of targeted synonyms in the emotion classification problem, which affects the accuracy of the emotion classification model.
By obtaining the words to be replaced from the original text and determining their target synonyms based on the pre-established test set vocabulary list, the replacement is obtained to ensure that the enhanced text has a certain correlation with the test text.
The data enhancement quality is improved, and the problem of the lack of synonyms and thesaurus targeted impact on the accuracy of the emotional classification model is solved, and the enhanced text is more correlated with the test text.
Smart Images
Figure CN112183074B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to computer technology, and more particularly to a data enhancement method, device, equipment and medium. Background Art
[0002] Sentiment classification is an important research direction in Natural Language Processing (NLP). Sentiment classification refers to dividing text into two or more types according to the meaning and emotional information expressed in the text. The training of sentiment classification models requires a large number of training sentences. When the training sentences are insufficient, data enhancement is often required.
[0003] Existing NLP data enhancement technologies include random insertion, random deletion, random swap, synonym replacement, and back translation. Among them, a common problem of random insertion, random deletion, and random swap methods is that they will change the semantics of the original text, so these methods are not suitable for data enhancement when dealing with sentiment classification problems; in addition, although synonym replacement and back translation methods can better retain the semantics of the original text, the alternative word library when selecting synonyms is the word library corresponding to the entire dictionary, which is not targeted and affects the accuracy of sentiment classification model classification. Summary of the invention
[0004] The embodiments of the present invention provide a data enhancement method, device, equipment and medium, which combine test set data so that the obtained enhanced text has a certain correlation with the test text, thereby improving the quality of data enhancement.
[0005] In a first aspect, an embodiment of the present invention provides a data enhancement method, the method comprising:
[0006] Get a set number of words to be replaced from the original text;
[0007] Determining a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set;
[0008] The target synonym is used to replace the word to be replaced, thereby obtaining an enhanced text.
[0009] In a second aspect, an embodiment of the present invention further provides a data enhancement device, the device comprising:
[0010] A to-be-replaced word acquisition module is used to acquire a set number of to-be-replaced words from the original text;
[0011] A target synonym determination module, configured to determine a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set;
[0012] The enhanced text acquisition module is used to replace the to-be-replaced word with the target synonym to obtain the enhanced text.
[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, including:
[0014] one or more processors;
[0015] A memory for storing one or more programs;
[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data enhancement method provided by any embodiment of the present invention.
[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data enhancement method provided by any embodiment of the present invention.
[0018] The technical solution of the embodiment of the present invention first obtains a set number of words to be replaced from the original text, then determines the target synonyms of each word to be replaced according to a pre-established test set vocabulary, and finally replaces the words to be replaced by the target synonyms to obtain enhanced text. This solves the problem in the prior art that when selecting synonyms, the alternative vocabulary corresponds to the vocabulary of the entire dictionary, which is not targeted and affects the accuracy of the sentiment classification model. By combining the test set data distribution, the obtained enhanced text has a certain correlation with the test text, thereby improving the quality of data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 is a flow chart of a data enhancement method in Embodiment 1 of the present invention;
[0020] Figure 2 is a flow chart of a data enhancement method in Embodiment 2 of the present invention;
[0021] Figure 3 is a flow chart of a data enhancement method in Embodiment 3 of the present invention;
[0022] Figure 4 is a structural schematic diagram of a data enhancement device in Embodiment 4 of the present invention;
[0023] Figure 5 It is a structural schematic diagram of a device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION
[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0025] Embodiment 1
[0026] Figure 1 This is a flow chart of a data enhancement method in Example 1 of the present invention. The technical solution of this embodiment is applicable to the case of performing synonym filtering through a pre-established test set vocabulary. The method can be executed by a data enhancement device, which can be implemented by software and / or hardware and can be integrated in various general-purpose computer devices.
[0027] To facilitate understanding, we first explain and analyze the commonly used data augmentation methods. When there is insufficient data in the test set, it is difficult to train a sentiment classification model with good classification results. Therefore, data augmentation is required based on the original text.
[0028] Commonly used data enhancement methods include random insertion, random deletion, random exchange, synonym replacement and back translation. Among them, the random insertion method refers to randomly finding a word in the original text that does not belong to the stop word set, and finding its random synonym, inserting the synonym into a random position of the sentence, repeating it many times, and obtaining an enhanced text; random deletion refers to randomly removing words in the original text with a certain probability; random exchange refers to randomly selecting two words in the original text and exchanging their positions, repeating it many times. A common problem of the above three data enhancement methods is that they will change the semantics of the original text. For sentiment classification problems, the sentiment tendency of a sentence may be determined by a few key words. If they happen to be inserted, deleted and exchanged, the sentiment tendency of the whole sentence may change. Therefore, the above three data enhancement methods are not applicable in the field of sentiment classification.
[0029] The synonym replacement method is the most commonly used NLP data enhancement method. The main operation is to randomly select a set number of words that do not belong to the stop word set from the original text, and randomly select their synonyms to replace them; the back translation method refers to translating a sentence, such as Chinese, into another language, such as French, and then translating the French sentence into Chinese, or other languages such as English and Japanese. Compared with the previous three methods, these two methods can better preserve the semantics of the original text. However, there are still some shortcomings. When they select synonyms for word replacement or translation, their alternative word library is the word library corresponding to the entire dictionary, and it is not combined with the word information in the test set, so it is not targeted.
[0030] The data enhancement method in this embodiment specifically includes the following steps:
[0031] Step 110: Obtain a set number of words to be replaced from the original text.
[0032] Among them, the original text refers to the text originally existing in the test set. For example, the text "We are like dandelions, and I also pray that we can fly to the same land with you" originally existing in the test set; the words to be replaced are selected from the original text and will be replaced by their synonyms. For example, in the original text "We are like dandelions, and I also pray that we can fly to the same land with you", "pray" is selected as the word to be replaced, and its synonyms can be selected later for word replacement to achieve data augmentation.
[0033] In this embodiment, in order to perform data augmentation by using the method of synonym replacement, first, a set number of words to be replaced are selected from the original text. Specifically, first, the original text is tokenized to obtain at least one word, then the words belonging to the stop words are removed from the at least one word, and finally, a set number of words to be replaced are selected from the remaining words. Among them, the number of words to be replaced can be a preset fixed value, for example, 1, or it can be determined according to the number of remaining words. For example, if the replacement ratio is 10%, when the number of remaining words is 10, the number of words to be replaced is 1.
[0034] Among them, stop words refer to certain words or phrases that are automatically filtered out before or after processing natural language data (or text) in information retrieval to save storage space and improve search efficiency. These words are usually some modal particles, adverbs, prepositions, conjunctions, etc., and usually have no clear meaning by themselves. For example, common words like "de" (的) and "zai" (在).
[0035] Step 120: Determine the target synonyms of each word to be replaced according to the pre-established test set word list, where the test set word list contains at least one word in the test set.
[0036] Among them, the test set word list is a set of words containing at least one word in the test set.
[0037] In this embodiment, before performing data augmentation, a test set word list will be generated in advance according to the test set data, and the test set word list contains the words in the test set data. In order to make the finally obtained augmented text have a certain correlation with the test text contained in the test set, the target synonyms of each word to be replaced are determined according to the pre-established test set word list.
[0038] Exemplarily, the synonyms of the word to be replaced can be queried in the test set vocabulary. When there are multiple synonyms, one is randomly selected as the target synonym for the current word to be replaced. When there is one synonym, the synonym is used as the target synonym. When the synonym of the current word to be replaced does not exist in the test set vocabulary, at least one synonym of the word to be replaced is searched in the complete vocabulary library, and a synonym is randomly selected as the target synonym.
[0039] Step 130: Replace the word to be replaced with the target synonym to obtain enhanced text.
[0040] In this embodiment, after determining the target synonym, the target synonym is used to replace the corresponding word to be replaced to obtain the enhanced text. For example, the original text is "We are like dandelions, and I also pray that I can fly to the same land with you", the word to be replaced is "prayer", and the target synonym is "prayer", then the enhanced text is "We are like dandelions, and I also pray that I can fly to the same land with you".
[0041] The technical solution of the embodiment of the present invention first obtains a set number of words to be replaced from the original text, then determines the target synonyms of each word to be replaced according to a pre-established test set vocabulary, and finally replaces the words to be replaced by the target synonyms to obtain enhanced text. This solves the problem in the prior art that when selecting synonyms, the alternative vocabulary corresponds to the vocabulary of the entire dictionary, which is not targeted and affects the accuracy of the sentiment classification model. By combining the test set data, the obtained enhanced text has a certain correlation with the test text, thereby improving the quality of data enhancement.
[0042] Embodiment 2
[0043] Figure 2 This is a flow chart of a data enhancement method in Embodiment 2 of the present invention. This embodiment further refines the above embodiment and provides specific steps for obtaining a set number of words to be replaced from the original text. Figure 2 A data enhancement method provided in the second embodiment of the present invention is described, which includes the following steps:
[0044] Step 210: Perform word segmentation processing on the original text to obtain at least one initial word.
[0045] Among them, word segmentation is the process of recombining continuous text sequences into word sequences according to certain specifications.
[0046] In this embodiment, in order to perform data enhancement on the original text and obtain more enhanced text for sentiment classification model training, the original text is firstly segmented to obtain at least one initial word corresponding to the original text.
[0047] For example, the initial text is "We are like dandelions, and I pray that I can fly to the same land with you", and the result after word segmentation is "We are like dandelions,' and I pray that I can fly to the same land with you."
[0048] Optionally, the original text is segmented to obtain at least one initial word, including:
[0049] The Jieba word segmentation method is used to segment the original text to obtain at least one initial word.
[0050] In this optional embodiment, a word segmentation method is provided, specifically, using the Jieba word segmentation method to perform word segmentation processing on the original text to obtain at least one initial word. Among them, the Jieba word segmentation method is a word segmentation method, which is widely used for its advantages of fast word segmentation speed and high accuracy.
[0051] Step 220: Delete the stop words contained in at least one initial word to obtain at least one target word.
[0052] In this embodiment, after the original text is segmented to obtain at least one initial word, stop words that have no practical meaning in the at least one initial word are removed to obtain at least one target word. Exemplarily, the original text is segmented to obtain at least one initial word including "we are like dandelions,',' I also pray that I can fly to the same piece of land with you", and the stop words that have no practical meaning are removed, and the remaining words are "dandelions, pray, fly to, a piece of land", and the remaining words are used as target words.
[0053] Step 230: Select a set number of words to be replaced from at least one target word by setting rules.
[0054] In this embodiment, according to the preset selection rule of words to be replaced, a set number of words are selected as words to be replaced from at least one target word obtained in step 220. Exemplarily, two target words are randomly selected from at least one target word as words to be replaced; alternatively, replacement words are selected according to the number of target words and a set ratio, for example, 10% of the target words are randomly selected as words to be replaced.
[0055] Optionally, a set number of words to be replaced are selected from at least one target word by setting a rule, including:
[0056] Determine the word quantity of the target word corresponding to the original text, and multiply the word quantity by the preset word selection ratio to obtain the word selection quantity;
[0057] From the target words, select the selected word number of target words as the words to be replaced.
[0058] In this optional embodiment, a specific method is provided for selecting a set number of words to be replaced from at least one target word by setting rules. First, the number of words to be replaced is determined, specifically, the number of words in the target words corresponding to the original text is determined, and the number of words is multiplied by a preset word selection ratio to determine the number of selected words. Then, the number of target words to be replaced is randomly selected from the target words as the words to be replaced.
[0059] Exemplarily, at least one target word obtained is "dandelion, pray, fly away, a piece of land", and the preset word selection ratio is 10%. Then, the number of target words is first determined to be 5, and the number of target words is multiplied by 10%, and the number of selected words obtained is 0.5. At this time, since the number of selected words must be an integer, 1 word is directly selected as the target word, that is, among the above target words, a word is randomly selected as the word to be replaced.
[0060] Of course, after determining the number of words to be selected, you can also select target words of a set type as the words to be replaced according to the actual situation. For example, you can select verbs or adjectives as the words to be replaced. There is no specific limitation here.
[0061] Step 240: Determine a target synonym for each word to be replaced based on a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set.
[0062] Step 250: Replace the word to be replaced with the target synonym to obtain enhanced text.
[0063] The technical solution of this embodiment first performs word segmentation processing on the original text to obtain at least one initial word, then deletes the stop words contained in the at least one initial word to obtain at least one target word, and selects a set number of to-be-replaced words from the at least one target word by setting rules, and further, determines the target synonyms of each to-be-replaced word according to a pre-established test set word list, and finally replaces the to-be-replaced word with the target synonyms to obtain an enhanced text, which solves the problem in the prior art that when selecting synonyms, the alternative word library corresponds to the word library of the entire dictionary and is not targeted, and improves the quality of data enhancement by combining the test set data.
[0064] Embodiment 3
[0065] Figure 3This is a flow chart of a data enhancement method in Embodiment 3 of the present invention. This embodiment further refines the above embodiment and provides specific steps before obtaining a set number of words to be replaced from the original text, and specific steps for determining the target synonyms of each word to be replaced according to a pre-established test set vocabulary. Figure 3 A data enhancement method provided in Embodiment 3 of the present invention is described, and includes the following steps:
[0066] Step 310: Perform word segmentation on the test text contained in the test set to obtain a test word list.
[0067] In this embodiment, in order to establish a test set vocabulary, multiple test texts included in the test set are first segmented to obtain a test word list, wherein the test word list contains all the words in the test set, and these words can represent the application scenarios of the test texts.
[0068] Step 320: Remove duplicate test words in the test word list to obtain a test set word list.
[0069] In this embodiment, the obtained test word list may contain many repeated test words. At this time, it is necessary to deduplicate the test word list to obtain a test set word list. Specifically, the test word list can be input into the unique function to obtain a test set word list that does not contain repeated test words.
[0070] Step 330: Obtain a set number of words to be replaced from the original text.
[0071] Step 340: Determine a target synonym for each word to be replaced based on a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set.
[0072] Optionally, according to a pre-established test set vocabulary, target synonyms for each word to be replaced are determined, including:
[0073] Searching the complete synonym set corresponding to each word to be replaced in the complete word library, wherein the complete synonym set contains at least one synonym of the word to be replaced;
[0074] Through the test set vocabulary, the complete synonym set is filtered to obtain the target synonym set;
[0075] The target synonym of the word to be replaced is determined in the target synonym set according to the set rules.
[0076] In this optional embodiment, a specific method is provided for determining the target synonyms of each word to be replaced based on a pre-established test set vocabulary. First, all synonyms corresponding to the word to be replaced are queried in the complete vocabulary library to form a complete synonym set. Then, the complete synonym set is filtered through the test set vocabulary to filter out the synonyms that are not in the test set vocabulary to obtain the target synonym set. Finally, a target synonym is randomly selected from the target synonym set as the target synonym of the word to be replaced.
[0077] Exemplarily, the word to be replaced is "prayer". The complete synonym set corresponding to the word to be replaced is queried in the complete vocabulary library, including "prayer, worship, worship, chanting, religious ceremony, blessing, confession, prayer, dedication". Then, the complete synonym set is filtered according to the test set vocabulary list to filter out the words not included in the test set vocabulary list, and the target synonym set is obtained as "prayer, worship, blessing, confession, dedication". Finally, a word can be randomly selected from the target synonym set as the target synonym of the word to be replaced, for example, "prayer" is selected as the target synonym.
[0078] Among them, the complete word library corresponds to the word library of the entire dictionary. Therefore, the synonyms contained in the complete synonym set queried in the complete word library are not necessarily the words required by the application scenario of the test set. Therefore, using the test set word list to filter the complete synonym set can make the final target synonyms related to the application scenario of the test set, which can improve the quality of data enhancement.
[0079] Optionally, according to a pre-established test set vocabulary, target synonyms for each word to be replaced are determined, including:
[0080] Search the test set vocabulary for the test synonyms corresponding to each word to be replaced;
[0081] When there is at least one test synonym in the test set vocabulary, a target synonym of the word to be replaced is determined from the at least one test synonym according to a set rule;
[0082] When the test synonym does not exist in the test set vocabulary, the complete synonym set corresponding to the word to be replaced is queried from the complete vocabulary library, and the target synonym of the word to be replaced is determined from the complete synonym set by setting rules.
[0083] In this optional embodiment, another specific method for determining the target synonym of each word to be replaced based on a pre-established test set vocabulary is provided. First, the test synonym corresponding to the word to be replaced is queried in the test set vocabulary. When there is at least one test synonym in the test set vocabulary, one can be randomly selected from the at least one test synonym as the target synonym for the word to be replaced. Of course, the first queried test synonym can also be selected as the target synonym in the query order, which is not specifically limited here. When the test synonym does not exist in the test set vocabulary, the complete synonym set corresponding to the word to be replaced is further queried from the complete vocabulary, and the target synonym of the word to be replaced is determined from the complete synonym set by setting rules, for example, a synonym is randomly selected from the complete synonym set as the target synonym.
[0084] Step 350: Replace the word to be replaced with the target synonym to obtain enhanced text.
[0085] The technical solution of this embodiment first performs word segmentation processing on the test text contained in the test set to obtain a test word list, and removes repeated test words in the test word list to obtain a test set word list, then obtains a set number of words to be replaced from the original text, and determines the target synonyms of each word to be replaced according to the pre-established test set word list, and finally replaces the words to be replaced with the target synonyms to obtain an enhanced text, and filters the complete synonym set through the test set word list to improve the quality of data enhancement.
[0086] Embodiment 4
[0087] Figure 4 This is a structural diagram of a data enhancement device provided in Embodiment 4 of the present invention. The data enhancement device includes: a to-be-replaced word acquisition module 410, a target synonym determination module 420 and an enhanced text acquisition module 430.
[0088] The to-be-replaced word acquisition module 410 is used to acquire a set number of to-be-replaced words from the original text;
[0089] A target synonym determination module 420, configured to determine a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set;
[0090] The enhanced text acquisition module 430 is used to replace the to-be-replaced word with the target synonym to obtain enhanced text.
[0091] The technical solution of the embodiment of the present invention first obtains a set number of words to be replaced from the original text, then determines the target synonyms of each word to be replaced according to a pre-established test set vocabulary, and finally replaces the words to be replaced by the target synonyms to obtain enhanced text. This solves the problem in the prior art that when selecting synonyms, the alternative vocabulary corresponds to the vocabulary of the entire dictionary, which is not targeted and affects the accuracy of the sentiment classification model. By combining the test set data distribution, the obtained enhanced text has a certain correlation with the test text, thereby improving the quality of data enhancement.
[0092] Optionally, the to-be-replaced word acquisition module 410 includes:
[0093] An initial word acquisition unit, used for performing word segmentation processing on the original text to obtain at least one initial word;
[0094] a target word acquisition unit, configured to delete stop words contained in the at least one initial word to obtain at least one target word;
[0095] The to-be-replaced word acquisition unit is used to select a set number of to-be-replaced words from the at least one target word by setting rules.
[0096] Optionally, the target synonym determination module 420 includes:
[0097] A complete synonym set acquisition unit, used for searching the complete synonym set corresponding to each of the words to be replaced in the complete word library, wherein the complete synonym set contains at least one synonym of the word to be replaced;
[0098] A target synonym set acquisition unit is used to filter the complete synonym set through a test set vocabulary to obtain a target synonym set;
[0099] The target synonym determination unit is used to determine the target synonym of the word to be replaced in the target synonym set according to a set rule.
[0100] Optionally, the initial word acquisition unit is specifically used to:
[0101] The original text is segmented using the Jieba word segmentation method to obtain at least one initial word.
[0102] Optionally, the to-be-replaced word acquisition unit is specifically used to:
[0103] Determine the word quantity of the target words corresponding to the original text, and multiply the word quantity by a preset word selection ratio to obtain the word selection quantity;
[0104] From the target words, select the selected number of target words as words to be replaced.
[0105] Optionally, the data enhancement device further includes:
[0106] A test word list acquisition module is used to perform word segmentation processing on the test text contained in the test set before obtaining a set number of words to be replaced from the original text to obtain a test word list;
[0107] The test set vocabulary acquisition module is used to remove repeated test words in the test word list to obtain the test set vocabulary.
[0108] Optionally, the target synonym determination module 420 includes:
[0109] A test synonym query unit, used for querying the test synonym corresponding to each of the words to be replaced in the test set vocabulary;
[0110] A first target synonym determination unit, configured to determine a target synonym of the word to be replaced from the at least one test synonym according to a set rule when the test set vocabulary contains at least one test synonym;
[0111] The second target synonym determination unit is used to query the complete synonym set corresponding to the word to be replaced from the complete word library when the test synonym does not exist in the test set vocabulary, and determine the target synonym of the word to be replaced from the complete synonym set by setting rules.
[0112] The data enhancement device provided in the embodiment of the present invention can execute the data enhancement method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0113] Embodiment 5
[0114] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention is shown in FIG. Figure 5 As shown, the electronic device includes a processor 50 and a memory 51; the number of processors 50 in the device can be one or more. Figure 5 A processor 50 is taken as an example; the processor 50 and the memory 51 in the device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.
[0115] The memory 51, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to a data enhancement method in an embodiment of the present invention (for example, the to-be-replaced word acquisition module 410, the target synonym determination module 420 and the enhanced text acquisition module 430 in the data enhancement device). The processor 50 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 51, that is, implements the above-mentioned data enhancement method.
[0116] The method includes:
[0117] Get a set number of words to be replaced from the original text;
[0118] Determining a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set;
[0119] The target synonym is used to replace the word to be replaced, thereby obtaining an enhanced text.
[0120] The memory 51 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 51 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 51 may further include a memory remotely arranged relative to the processor 50, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0121] Embodiment 6
[0122] Embodiment 6 of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to perform a data enhancement method when executed by a computer processor, the method comprising:
[0123] Get a set number of words to be replaced from the original text;
[0124] Determining a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set;
[0125] The target synonym is used to replace the word to be replaced, thereby obtaining an enhanced text.
[0126] Of course, the computer executable instructions of the storage medium containing computer executable instructions provided in the embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the data enhancement method provided in any embodiment of the present invention.
[0127] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, an application server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0128] It is worth noting that in the embodiment of the above-mentioned data enhancement device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0129] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data enhancement method, characterized in that: include: Perform word segmentation on the test texts contained in the test set to obtain a test word list; The words in the test word list are used to characterize the application scenarios of the test text; Removing repeated test words from the test word list to obtain a test set word list; Get a set number of words to be replaced from the original text; Determining a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set; Replacing the word to be replaced by the target synonym to obtain an enhanced text; The step of determining the target synonyms of each of the words to be replaced based on a pre-established test set vocabulary includes: Searching the test set vocabulary for the test synonyms corresponding to the words to be replaced; When there is at least one test synonym in the test set vocabulary, determining a target synonym of the word to be replaced from the at least one test synonym according to a set rule; When the test synonym does not exist in the test set vocabulary, query the complete synonym set corresponding to the word to be replaced from the complete word library, and determine the target synonym of the word to be replaced from the complete synonym set by setting rules; Alternatively, a complete synonym set corresponding to each of the words to be replaced is searched in a complete word library, wherein the complete synonym set contains at least one synonym of the word to be replaced; The complete synonym set is filtered through the test set vocabulary to obtain a target synonym set; The target synonym of the word to be replaced is determined in the target synonym set according to the set rules.
2. The method according to claim 1, characterized in that: Get a set number of words to be replaced from the original text, including: Performing word segmentation processing on the original text to obtain at least one initial word; Deleting stop words contained in the at least one initial word to obtain at least one target word; By setting rules, a set number of words to be replaced are selected from the at least one target word.
3. The method according to claim 2, characterized in that Perform word segmentation on the original text to obtain at least one initial word, including: The original text is segmented using the Jieba word segmentation method to obtain at least one initial word.
4. The method according to claim 2, characterized in that: By setting rules, a set number of words to be replaced are selected from the at least one target word, including: Determine the word quantity of the target words corresponding to the original text, and multiply the word quantity by a preset word selection ratio to obtain the word selection quantity; From the target words, select the selected number of target words as words to be replaced.
5. A data enhancement device, characterized in that: include: A test word list acquisition module is used to perform word segmentation processing on the test text contained in the test set to obtain a test word list; The words in the test word list are used to characterize the application scenarios of the test text; A test set vocabulary acquisition module, used to remove repeated test words in the test word list to obtain a test set vocabulary; A to-be-replaced word acquisition module is used to acquire a set number of to-be-replaced words from the original text; A target synonym determination module, configured to determine a target synonym for each of the words to be replaced according to a pre-established test set vocabulary, wherein the test set vocabulary contains at least one word in the test set; An enhanced text acquisition module, used to replace the to-be-replaced word with the target synonym to obtain an enhanced text; Wherein, the target synonym determination module includes: A test synonym query unit, used for querying the test synonym corresponding to each of the words to be replaced in the test set vocabulary; A first target synonym determination unit, configured to determine a target synonym of the word to be replaced from the at least one test synonym according to a set rule when the test set vocabulary contains at least one test synonym; A second target synonym determination unit is used for, when the test synonym does not exist in the test set vocabulary, querying a complete synonym set corresponding to the word to be replaced from a complete word library, and determining a target synonym for the word to be replaced from the complete synonym set by setting rules; Alternatively, the target synonym determination module includes: A complete synonym set acquisition unit, used for searching the complete synonym set corresponding to each of the words to be replaced in the complete word library, wherein the complete synonym set contains at least one synonym of the word to be replaced; A target synonym set acquisition unit is used to filter the complete synonym set through a test set vocabulary to obtain a target synonym set; The target synonym determination unit is used to determine the target synonym of the word to be replaced in the target synonym set according to a set rule.
6. The device according to claim 5, characterized in that The to-be-replaced word acquisition module comprises: An initial word acquisition unit, used for performing word segmentation processing on the original text to obtain at least one initial word; a target word acquisition unit, configured to delete stop words contained in the at least one initial word to obtain at least one target word; The to-be-replaced word acquisition unit is used to select a set number of to-be-replaced words from the at least one target word by setting rules.
7. The device according to claim 6, characterized in that The initial word acquisition unit is specifically used for: The original text is segmented using the Jieba word segmentation method to obtain at least one initial word.
8. The device according to claim 6, characterized in that The to-be-replaced word acquisition unit is specifically used for: Determine the word quantity of the target words corresponding to the original text, and multiply the word quantity by a preset word selection ratio to obtain the word selection quantity; From the target words, select the selected number of target words as words to be replaced.
9. An electronic device, characterized in that: The device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data enhancement method as described in any one of claims 1-4.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data enhancement method as described in any one of claims 1 to 4 is implemented.