Unsupervised paraphrase text generation method, device and medium based on machine translation

Abstract

The present application discloses an unsupervised paraphrase text generation method, device and medium based on machine translation, which belongs to the field of text classification. The method mainly includes obtaining unannotated monolingual corpus exceeding a first preset number, and using it to train an unsupervised machine translation model; inputting source sentences exceeding a second preset number into the unsupervised machine translation model to obtain corresponding paraphrased sentences, and the source sentences and the corresponding paraphrased sentences constitute pseudo-paraphrased text sentence pairs; in each pseudo-paraphrased text sentence pair, calculating the similarity between the source sentence and the paraphrased sentence, filtering out pseudo-paraphrased text sentence pairs with similarities greater than a preset threshold, and obtaining a pseudo-paraphrased text sentence pair set; using the pseudo-paraphrased text sentence pair set, training a paraphrased text generation model, and using the paraphrased text generation model to process the input text to obtain paraphrased text. The present application only requires a large amount of unannotated monolingual corpus to complete the training, which greatly reduces the dependence on annotated data.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method, device and medium for generating unsupervised paraphrased text based on machine translation. Background Art

[0002] The motivation for this scheme comes from the ability of back-translation to generate a large number of pseudo-parallel sentence pairs. Back-translation means first translating a sentence in language A, such as Chinese, into a corresponding sentence in language B, such as English, and then translating the sentence back to Chinese. In this way, the original sentence and the back-translated sentence form a pair of pseudo-parallel reported sentence pairs. For example, the original sentence is "I love China", which is translated into English as "I like China" and then translated back into Chinese as "I like China". "I love China" and "I like China" are pseudo-parallel reported sentence pairs generated by back-translation. This method can generate a large number of pseudo-parallel reported sentence pairs, and finally these sentence pairs can be used to train a reported text generation model.

[0003] However, there is a problem with this approach: training machine translation models also requires bilingual parallel corpora, and it is difficult to obtain a large amount of bilingual parallel corpora. Therefore, this solution uses the "unsupervised machine translation" method to train machine translation models. Only a large amount of monolingual non-parallel corpora are needed to train a well-performing machine translation model. Although the effect is slightly worse than that of a machine translation model trained on bilingual parallel corpora, the ultimate goal of this application is to generate a large number of pseudo-parallel text sentence pairs to train a paraphrase text generation model, so unsupervised machine translation can also meet the expected effect well. Summary of the invention

[0004] In view of the problems in the prior art that it is difficult to obtain bilingual parallel corpora required for machine translation models and that they are highly dependent on annotated data, the present application mainly provides a method, device and medium for generating unsupervised paraphrased text based on machine translation.

[0005] In order to achieve the above-mentioned purpose, a technical solution adopted by the present application is to provide an unsupervised paraphrase text generation method based on machine translation, which comprises:

[0006] Obtaining unlabeled monolingual corpora exceeding a first preset amount, and using the unlabeled monolingual corpora to train an unsupervised machine translation model;

[0007] Inputting source sentences exceeding a second preset number into an unsupervised machine translation model to obtain corresponding paraphrased sentences, wherein the source sentences and the corresponding paraphrased sentences constitute pseudo-paraphrased text sentence pairs;

[0008] In each pseudo-paragraphed text sentence pair, the similarity between the source sentence and the paragraphed sentence is calculated, and the pseudo-paragraphed text sentence pairs with similarity greater than a preset threshold are filtered out to obtain a set of pseudo-paragraphed text sentence pairs; and

[0009] The pseudo-paragraph text sentence pair set is used to train the paragraph text generation model, and the paragraph text generation model is used to process the input text to obtain the paragraph text.

[0010] Another technical solution adopted by the present application is to provide an unsupervised paraphrase text generation device based on machine translation, which comprises:

[0011] A translation model training module, which is used to obtain unlabeled monolingual corpora exceeding a first preset amount and use the unsupervised machine translation model to train the unsupervised machine translation model;

[0012] a pseudo-paragraph text sentence pair generation module, which is used to input source sentences exceeding a second preset number into an unsupervised machine translation model to obtain corresponding paragraph sentences, wherein the source sentences and the corresponding paragraph sentences constitute pseudo-paragraph text sentence pairs;

[0013] for calculating the similarity between the source sentence and the reported sentence in each pseudo-reported text sentence pair, filtering out pseudo-reported text sentence pairs whose similarity is greater than a preset threshold, and obtaining a set of pseudo-reported text sentence pairs; and

[0014] The reported text generation model training module is used to train the reported text generation model using the pseudo reported text sentence pair set, and use the reported text generation model to process the input text to obtain the reported text.

[0015] Another technical solution adopted by the present application is: providing a computer-readable storage medium storing computer instructions, which are operated to execute the unsupervised paraphrase text generation method based on machine translation in Solution 1.

[0016] Another technical solution adopted by the present application is: providing a computer device, which includes a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are operated to execute the unsupervised paraphrase text generation method based on machine translation in solution one.

[0017] The beneficial effects that can be achieved by the technical solution of the present application are as follows: the present application designs a method, device and medium for unsupervised paraphrase text generation based on machine translation. The method generates a large number of high-quality paraphrase text pairs in an unsupervised manner through reverse machine translation, trains a paraphrase text generation model on the pseudo paraphrase text pairs, and obtains the final paraphrase text. This method does not require any annotated bilingual corpus, but only requires a large amount of unannotated monolingual corpus to complete the training, greatly reducing the dependence on annotated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a schematic diagram of a specific implementation method of an unsupervised paraphrase text generation method based on machine translation of the present application;

[0020] Figure 2 is a schematic diagram of a specific example of an unsupervised paraphrase text generation method based on machine translation in the present application;

[0021] Figure 3 It is a schematic diagram of a specific implementation of an unsupervised paraphrase text generation device based on machine translation in the present application.

[0022] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present application.

[0024] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0025] This application trains a paraphrased text generation model unsupervised based on machine translation. First, this application will train a reverse machine translation model on a large amount of non-parallel monolingual corpora. The reverse machine translation model is responsible for translating the input source sentence into another language and then translating it back. After training, a large number of pseudo-paraphrased text pairs can be generated using this reverse machine translation model. Then, the obtained pseudo-paraphrased text pairs are filtered to exclude the pairs with lower quality. Finally, the remaining high-quality pseudo-paraphrased text pairs are used to train the final paraphrased text generation model. This application does not require bilingual parallel corpora, only non-parallel monolingual corpora, and can obtain a paraphrased text generation model with good results. This application is trained in such a completely unsupervised method, which greatly reduces the dependence on labeled data and can also achieve good results.

[0026] Training a translation model using labeled data essentially means that given an input source sentence, the translation model only needs to output the labeled result. For example, if the labeled data is "我爱你" ---> "I love you", then given the input source sentence "我爱你", the translation model only needs to calculate the maximum probability of "I love you". However, in the unlabeled data, there is no information about what the true translation of the source sentence "我爱你" is. At this time, two machine translation models in different directions need to be trained, such as one direction from Chinese to English and the other direction from English to Chinese. First, the source sentence "我爱你" is sent into the machine translation model from Chinese to English to obtain a translation result, and then this obtained translation result is sent into the machine translation model from English to Chinese to get another translation result. At this time, the goal is to make the result of the English-to-Chinese translation as close as possible to "我爱你", which is the input of the original Chinese-to-English machine translation model. This method is called "back-translation", and using this method, machine translation without labeled data can be achieved.

[0027] The following will specifically describe the technical solutions of this application and how the technical solutions of this application solve the above technical problems through specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0028] Figure 1 A specific implementation manner of an unsupervised paraphrased text generation method based on machine translation in this application is shown.

[0029] In Figure 1 the specific implementation manner shown, the unsupervised paraphrased text generation method based on machine translation mainly includes:

[0030] Step S101, obtaining unlabeled monolingual corpora exceeding a first preset amount, and using the unlabeled monolingual corpora to train an unsupervised machine translation model;

[0031] Step S102: inputting source sentences exceeding a second preset number into an unsupervised machine translation model to obtain corresponding reported sentences, wherein the source sentences and the reported sentences constitute pseudo-reported text sentence pairs.

[0032] In this embodiment, the unsupervised machine translation model trained with unlabeled monolingual corpus has the ability to generate a large number of pseudo-parallel reported sentence pairs by reverse translation, translating the source sentence of the first language, if the first language is Chinese, into the corresponding sentence of the second language, and reversely translating the corresponding sentence back to the first language to obtain the reported sentence, so that the original sentence and the reverse translated sentence constitute a pair of pseudo-parallel reported sentence pairs. The machine translation model obtained by training with a large amount of unlabeled monolingual corpus can obtain a large number of pseudo-reported text sentence pairs, making the pseudo-reported text sentence pairs easier to obtain.

[0033] Specifically, unlabeled monolingual corpora exceeding a first preset number are obtained in the corpus, where the first preset number is 100G, that is, the content size of the obtained unlabeled monolingual corpora exceeds 100G; source sentences exceeding a second preset number are obtained in the source sentence library, where the second preset number is 50G, that is, the content size of the obtained source sentences exceeds 50G.

[0034] In an optional embodiment of the present application, source sentences exceeding a second preset number are input into an unsupervised machine translation model to obtain reported sentences exceeding the second preset number, including: using the unsupervised machine translation model to translate the input source sentences into target sentences of a language different from that of the source sentences, and then translating the target sentences into reported sentences of the same language as the source sentences.

[0035] In this optional embodiment, the main function of the unsupervised machine translation model is to translate the source sentence in the first language into the target sentence in the second language, and reversely translate the target sentence in the second language into the reported sentence in the first language, thereby obtaining a large number of pseudo-reported text sentence pairs consisting of source sentences and reported sentences, which is convenient for training the reported text generation model.

[0036] exist Figure 1 In the specific implementation manner shown, the unsupervised paraphrase text generation method based on machine translation also includes:

[0037] Step S103, in each pseudo-paragraphed text sentence pair, calculating the similarity between the source sentence and the paragraphed sentence, filtering out pseudo-paragraphed text sentence pairs whose similarity is greater than a preset threshold, and obtaining a pseudo-paragraphed text sentence pair set;

[0038] Step S104, using the pseudo-paragraph text sentence pair set, training a paragraph text generation model, and using the paragraph text generation model to process the input text to obtain a paragraph text.

[0039] In this embodiment, pseudo-paragraph text sentence pairs with similarity greater than a preset threshold are filtered out, and pseudo-paragraph text sentence pairs with lower quality are excluded. Finally, the remaining high-quality pseudo-paragraph text pairs are used to train the final paragraph text generation model to obtain a paragraph text with better effect.

[0040] Preferably, the value of the preset threshold can be reasonably determined based on experience, wherein the preset threshold can be set to 0.7.

[0041] In an optional embodiment of the present application, in each pseudo-paraphrased text sentence pair, calculating the similarity between the source sentence and the paraphrased sentence further includes: determining the number of identical words and the number of different words in the source sentence and the paraphrased sentence; calculating the ratio of the number of identical words to the number of different words to determine the similarity, wherein the higher the ratio, the greater the similarity.

[0042] In this specific embodiment, the fewer the same words are in the source sentence and the reported sentence, and the more the different words are, the smaller the ratio between the two is, the lower the similarity is, the higher the quality of the pseudo-reported sentence pair is, and the better the effect of generating reported text for the trained reported text generation model is.

[0043] Figure 2 A specific example of an unsupervised paraphrase text generation method based on machine translation of the present application is shown.

[0044] exist Figure 2 In the specific example shown, a large-scale non-parallel monolingual corpus is used, and its collection is denoted as D U , train an unsupervised machine translation model, use the trained unsupervised machine translation model to generate a large number of pseudo-paraphrased text sentence pairs, whose set is denoted as D C The generation method is: the unsupervised machine translation model first translates the input source sentence into the target sentence of its corresponding language, and then translates the target sentence back to the original language to form a pair of pseudo-paraphrased text sentences with the input source sentence. Since this application requires the paraphrased text to be as different in form as possible, this solution filters out D in this step. C The part of the source sentence that contains the highest similarity between the source sentence and the paraphrase, which can be expressed by the following similarity:

[0045]

[0046] Among them, |s i ∩s j |It is a sentence i and jThe number of identical words, |s i ∪s j |It is a sentence i and j The number of different words. The greater the similarity, the more similar the two sentences are. The set of pseudo-paraphrased text sentence pairs after filtering is recorded as D' C , that is, the pseudo-paraphrased text sentence pair set. Finally, we only need to use D' C Just train a paraphrase text generation model.

[0047] In an optional embodiment of the present application, using a set of pseudo-paragraph text sentence pairs to train a paragraph text generation model further includes: using a predetermined amount of annotated data to train the paragraph text generation model so that it can perceive and learn the rules of paragraph text generation in a specific field.

[0048] In this specific embodiment, the step of annotated data training is optional. The paraphrase text generation model obtained by training with the pseudo paraphrase text sentence pair set can be directly used for paraphrase text generation. However, since it is trained completely in an unsupervised manner, it may not cover all types of input texts, such as texts in certain specific fields. Therefore, a small amount of annotated data can be further used to train the model, so that it can better perceive and learn the rules of paraphrase text generation in specific fields, and fully combine the advantages of supervised learning and unsupervised learning.

[0049] A predetermined amount of annotated data can be used to deepen the perception rules of the reported text generation model for learning reported text generation in specific fields, where the predetermined amount is 500M, that is, the content size of the annotated data used is 500M. By using a small amount of annotated data for processing, the perception rules of the reported text generation model for learning reported text generation in specific fields are deepened, rather than relying entirely on annotated data for model training, thereby greatly reducing the dependence on annotated data.

[0050] An unsupervised machine translation model is trained using more than 100G of unlabeled monolingual corpus. The unsupervised machine translation model has a back-translation function. The source sentence is input to obtain the corresponding paraphrase sentence, that is, it is translated into other languages ​​according to the language of the source sentence, and then translated back to the paraphrase sentence corresponding to the language of the source sentence. In this way, the source sentence and the paraphrase sentence constitute a pair of pseudo-parallel paraphrase sentence pairs. A large number of pseudo-parallel paraphrase sentence pairs can be used to obtain a paraphrase text generation model with better effect. This application is trained in a completely unsupervised way, which greatly reduces the dependence on labeled data and can also achieve better results.

[0051] Figure 3 A specific implementation of an unsupervised paraphrase text generation device based on machine translation is shown in the present application.

[0052] exist Figure 3In the specific implementation shown, the unsupervised paraphrase text generation device based on machine translation mainly includes:

[0053] Module 301, a translation model training module, which is used to obtain unlabeled monolingual corpora exceeding a first preset amount, and use the unlabeled monolingual corpora to train an unsupervised machine translation model;

[0054] Module 302, a pseudo-paragraph text sentence pair generation module, which is used to input source sentences exceeding a second preset number into an unsupervised machine translation model to obtain corresponding paragraph sentences, wherein the source sentences and the corresponding paragraph sentences constitute pseudo-paragraph text sentence pairs;

[0055] Module 302 is further configured to calculate the similarity between the source sentence and the reported sentence in each pseudo-reported text sentence pair, filter out pseudo-reported text sentence pairs with similarity greater than a preset threshold, and obtain a pseudo-reported text sentence pair set; and

[0056] Module 303 is a reported text generation model training module, which is used to train the reported text generation model using the pseudo reported text sentence pair set, and use the reported text generation model to process the input text to obtain the reported text.

[0057] In this specific implementation, a large amount of unlabeled monolingual corpora are obtained from the corpus, and the unsupervised machine translation model obtained by training is slightly inferior to the machine translation model trained on the bilingual parallel corpus, but it performs well and can meet the needs. A large number of pseudo-paragraphed text pairs are generated by using the unsupervised machine model, and the obtained pseudo-paragraphed text pairs are filtered to exclude pseudo-paragraphed text sentence pairs with lower quality, that is, pseudo-paragraphed text sentence pairs with high similarity, and the remaining high-quality pseudo-paragraphed text pairs are used to train the paragraphed text generation model. The obtained paragraphed text generation model has good effect and also greatly reduces the dependence of model training on labeled data.

[0058] The unsupervised reported text generation device based on machine translation provided in the present application can be used to execute the unsupervised reported text generation method based on machine translation described in any of the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0059] In a specific embodiment of the present application, each functional module in the unsupervised paraphrase text generation device based on machine translation of the present application can be directly in hardware, in a software module executed by a processor, or in a combination of the two.

[0060] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium.

[0061] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In an alternative, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0062] In another specific embodiment of the present application, a computer-readable storage medium stores computer instructions, and the computer instructions are operated to execute the unsupervised paraphrase text generation method based on machine translation in any embodiment.

[0063] In another specific embodiment of the present application, a computer device includes a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are operated to execute the unsupervised paraphrase text generation method based on machine translation in any one of the embodiments.

[0064] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structural transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating unsupervised paraphrased text based on machine translation, characterized in that: include: Acquire unlabeled monolingual corpora exceeding a first preset amount, and use the unlabeled monolingual corpora to train an unsupervised machine translation model so that the unsupervised machine translation model has the ability to reversely translate into sentences in the same language as the unlabeled monolingual corpora; Inputting source sentences exceeding a second preset number into the unsupervised machine translation model to obtain corresponding paraphrased sentences, wherein the source sentences and the corresponding paraphrased sentences constitute pseudo-paraphrased text sentence pairs, and the source sentences are in the same language as the unlabeled monolingual corpus, including using the unsupervised machine translation model to translate the input source sentences into target sentences in a language different from that of the source sentences, and then translating the target sentences into the paraphrased sentences in the same language as that of the source sentences; In each of the pseudo-reported text sentence pairs, the similarity between the source sentence and the reported sentence is calculated, and the pseudo-reported text sentence pairs whose similarity is greater than a preset threshold are filtered out to obtain a set of pseudo-reported text sentence pairs; and The pseudo-paragraph text sentence pair set is used to train a paragraph text generation model, and the paragraph text generation model is used to process an input text to obtain a paragraph text.

2. The unsupervised paraphrase text generation method based on machine translation according to claim 1, characterized in that: The method of training the reported text generation model by using the pseudo reported text sentence pair set further includes: The reported text generation model is trained using a predetermined amount of annotated data to enable it to perceive and learn the rules of reported text generation in a specific field.

3. The unsupervised paraphrase text generation method based on machine translation according to claim 1, characterized in that: The calculating the similarity between the source sentence and the reported sentence in each pseudo-reported text sentence pair further comprises: Determining the number of identical words and the number of different words in the source sentence and the reported sentence; The ratio of the number of identical words to the number of different words is calculated to determine the similarity, wherein the higher the ratio, the greater the similarity.

4. An unsupervised paraphrase text generation device based on machine translation, characterized in that: include: A translation model training module, which is used to obtain unlabeled monolingual corpora exceeding a first preset amount, and use the unlabeled monolingual corpora to train an unsupervised machine translation model so that the unsupervised machine translation model has the ability to reversely translate into sentences in the same language as the unlabeled monolingual corpora; a pseudo-paraphrased text sentence pair generation module, which is used to input source sentences exceeding a second preset number into the unsupervised machine translation model to obtain corresponding paraphrased sentences, wherein the source sentences and the corresponding paraphrased sentences constitute pseudo-paraphrased text sentence pairs, wherein the source sentences are in the same language as the unlabeled monolingual corpus, including using the unsupervised machine translation model to translate the input source sentences into target sentences in a language different from that of the source sentences, and then translating the target sentences into the paraphrased sentences in the same language as that of the source sentences; for calculating the similarity between the source sentence and the reported sentence in each of the pseudo-reported text sentence pairs, filtering out the pseudo-reported text sentence pairs whose similarity is greater than a preset threshold, and obtaining a set of pseudo-reported text sentence pairs; and The reported text generation model training module is used to train the reported text generation model using the pseudo reported text sentence pair set, and use the reported text generation model to process the input text to obtain the reported text.

5. The unsupervised paraphrase text generation device based on machine translation according to claim 4, characterized in that: In the pseudo-paraphrased text sentence pair generation module, the number of identical words and the number of different words in the source sentence and the paraphrased sentence are determined, the ratio of the number of identical words to the number of different words is calculated, and the similarity is determined, wherein the higher the ratio, the greater the similarity.

6. A computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are operated to execute the unsupervised paraphrase text generation method based on machine translation according to any one of claims 1 to 3.

7. A computer device comprising a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the method for generating unsupervised paraphrased text based on machine translation according to any one of claims 1 to 3.

Images