A method and device for selecting corpus

Abstract

The present invention provides a corpus screening method and device, wherein the method comprises the following steps: performing data cleaning on original corpus data to obtain candidate corpus data, performing word segmentation on sentences in the candidate corpus data to obtain a number of word units, calculating the ratio of the word unit list length and the word unit set length corresponding to the sentence, judging whether the sentence meets the preset conditions, and adding the sentence that meets the preset conditions to the corpus training database. The present invention judges whether the sentence meets the preset conditions by calculating the ratio of the word unit list length and the set length of the sentence, filters the sentences that do not meet the preset conditions, and adds the sentences that meet the preset conditions to the corpus training database for retraining of the neural network machine translation model, thereby reducing the probability of the same token being repeated in the sentence obtained after the neural network machine translation, improving the translation experience of the neural network machine, and improving the translation effect of the neural network machine.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a corpus screening method and device. Background Art

[0002] Machine translation, also known as automatic translation, is the process of using computers to convert one natural language (source language) into another natural language (target language). In order to achieve better results, machine translation requires the use of a large amount of corpus. When the corpus comes from the Internet, the quality of the corpus is often not guaranteed. In single or parallel corpora, there is a certain proportion of repeated occurrences of a single token (word unit, including characters, words, and symbols). The phenomenon of a token being repeated continuously seriously affects the experience of machine translation. Therefore, in view of the above situation, it is very necessary to start with the optimization of the corpus to improve the translation effect. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a corpus screening method and a corresponding corpus screening device that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, on the one hand, an embodiment of the present invention discloses a corpus screening method, comprising:

[0005] Get the original corpus data;

[0006] Performing data cleaning on the original corpus data to obtain candidate corpus data, wherein the candidate corpus data includes a plurality of sentences, and performing word segmentation on the plurality of sentences to obtain a plurality of word units;

[0007] Determine a word unit list and a word unit set according to the plurality of word units, wherein the word unit set is a set after removing repeated word units in the word unit list;

[0008] Calculate the word unit list length and word unit set length corresponding to the sentence, wherein the word unit list length is the number of the word units in the sentence, and the word unit set length is the number of the word units after removing repeated word units in the word unit list;

[0009] Calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and determining whether the sentence meets a preset condition;

[0010] Filter out sentences that do not meet the preset conditions, and add sentences that meet the preset conditions to the corpus training database.

[0011] Furthermore, the candidate corpus data includes a single corpus sentence, and the calculating of the word unit list length and the word unit set length of the sentence includes:

[0012] Calculate the first word unit list length and the first word unit set length of the monolingual sentence, where the first word unit list length is the number of the word units in the monolingual sentence, and the first word unit set length is the number of the first word units after removing repeated word units in the first word unit list.

[0013] Further, the calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judging whether the sentence meets the preset condition comprises:

[0014] Calculating a first word segmentation length ratio of the single language sentence, where the first word segmentation length ratio is a ratio of the first word unit list length to the first word unit set length;

[0015] It is determined whether the first word segmentation length ratio is less than a preset first threshold.

[0016] Furthermore, the candidate corpus data includes parallel corpus sentences, the parallel corpus sentences include source language sentences and target language sentences, and the calculating of the word unit list length and word unit set length corresponding to the sentences includes:

[0017] Calculating a second word unit list length and a second word unit set length of the source language sentence, wherein the second word unit list length is the number of the word units in the source language sentence, and the second word unit set length is the number of the second word units after removing repeated word units in the second word unit list;

[0018] The third word unit list length and the third word unit set length of the target language sentence are calculated, wherein the third word unit list length is the number of the word units in the target language sentence, and the third word unit set length is the number of the third word units after removing repeated word units in the third word unit list.

[0019] Further, the calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judging whether the sentence meets the preset condition comprises:

[0020] Calculating a second word segmentation length ratio of the source language sentence, where the second word segmentation length ratio is a ratio of the second word unit list length to the second word unit set length;

[0021] Calculating a third word segmentation length ratio of the target language sentence, wherein the third word segmentation length ratio is a ratio of the third word unit list length to the third word unit set length;

[0022] Calculating a fourth word segmentation length ratio of the parallel corpus sentence, wherein the fourth word segmentation length ratio is a ratio of the second word unit list length to the third word unit list length;

[0023] It is determined whether the second word segmentation length ratio is less than a preset second threshold, whether the third word segmentation length ratio is less than a preset third threshold, and whether the fourth word segmentation length ratio is within a preset fourth threshold interval.

[0024] On the other hand, the present invention also provides a corpus screening device, comprising:

[0025] The original corpus data acquisition module is used to acquire the original corpus data;

[0026] A first word unit acquisition module is used to perform data cleaning on the original corpus data to obtain candidate corpus data, wherein the candidate corpus data includes a plurality of sentences, and segment the plurality of sentences to obtain a plurality of word units;

[0027] A second word unit acquisition module, configured to determine a word unit list and a word unit set according to the plurality of word units, wherein the word unit set is a set after repeated word units in the word unit list are removed;

[0028] A third word unit acquisition module, used to calculate the word unit list length and word unit set length corresponding to the sentence, wherein the word unit list length is the number of the word units in the sentence, and the word unit set length is the number of the word units after removing repeated word units in the word unit list;

[0029] A judgment module, used for calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judging whether the sentence meets the preset conditions;

[0030] The corpus training database acquisition module is used to filter out sentences that do not meet the preset conditions and add sentences that meet the preset conditions to the corpus training database.

[0031] Furthermore, the candidate corpus data includes a single corpus sentence, and the third word unit acquisition module includes:

[0032] The first word unit acquisition submodule is used to calculate the first word unit list length and the first word unit set length of the monolingual sentence, wherein the first word unit list length is the number of the word units in the monolingual sentence, and the first word unit set length is the number of the first word units after removing repeated word units in the first word unit list.

[0033] Furthermore, the judging module includes:

[0034] A first calculation submodule, used for calculating a first word segmentation length ratio, where the first word segmentation length ratio is a ratio of the first word unit list length to the first word unit set length;

[0035] The first judgment submodule is used to judge whether the first word segmentation length ratio is less than a preset first threshold.

[0036] Furthermore, the candidate corpus data includes parallel corpus sentences, the parallel corpus sentences include source language sentences and target language sentences, and the third word unit acquisition module includes:

[0037] A second word unit acquisition submodule, configured to calculate a second word unit list length and a second word unit set length of the source language sentence, wherein the second word unit list length is the number of the word units in the source language sentence, and the second word unit set length is the number of the second word units after removing repeated word units in the second word unit list;

[0038] The third word unit acquisition submodule is used to calculate the third word unit list length and the third word unit set length of the target language sentence, wherein the third word unit list length is the number of the word units in the target language sentence, and the third word unit set length is the number of the third word units after removing repeated word units in the third word unit list.

[0039] Furthermore, the judging module includes:

[0040] A second calculation submodule, used for calculating a second word segmentation length ratio, where the second word segmentation length ratio is a ratio of the second word unit list length to the second word unit set length;

[0041] A second judgment submodule, used to judge whether the second word segmentation length ratio is less than a preset second threshold;

[0042] A third calculation submodule, used for calculating a third word segmentation length ratio, wherein the third word segmentation length ratio is a ratio of the third word unit list length to the third word unit set length;

[0043] A third judgment submodule, used to judge whether the third word segmentation length ratio is less than a preset third threshold;

[0044] a fourth calculation submodule, configured to calculate a fourth word segmentation length ratio, wherein the fourth word segmentation length ratio is a ratio of the second word unit list length to the third word unit list length;

[0045] The fourth judgment submodule is used to judge whether the fourth word segmentation length ratio is within a preset fourth threshold range.

[0046] On the other hand, an embodiment of the present invention further provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the steps of the corpus screening method when executed by the processor.

[0047] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the corpus screening method are implemented.

[0048] The embodiments of the present invention include the following advantages: the present invention performs data cleaning on original corpus data to obtain candidate corpus data, performs word segmentation on sentences in the candidate corpus data to obtain a number of word units, calculates the ratio of the word unit list length and the word unit set length corresponding to the sentence, determines whether the sentence meets preset conditions, adds the sentences that meet the preset conditions to the corpus training database for retraining of the neural network machine translation model, reduces the probability of the same token repeating continuously in the sentences obtained after the neural network machine translation, improves the translation experience of the neural network machine, and improves the translation effect of the neural network machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a method for selecting a corpus provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of translating English into Chinese in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of translating Chinese into English in an embodiment of the present invention;

[0052] Figure 4 A structural block diagram of a corpus screening device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Figure 1 A flowchart of a method for selecting corpus provided in an embodiment of the present invention. After selection by the method, corpus meeting preset conditions can be applied to the training of a neural network machine translation model. The method includes the following steps:

[0055] Step 101: Obtain original corpus data;

[0056] In this embodiment, the original corpus data may come from the user's candidate corpus, or the corresponding original corpus may be collected through the network. This application does not specifically limit the source of the original corpus data.

[0057] Step 102: clean the original corpus data to obtain candidate corpus data, wherein the candidate corpus data includes a plurality of sentences, and segment the plurality of sentences to obtain a plurality of word units;

[0058] Data cleaning of the original corpus data refers to unifying the language standards in the original corpus data, such as converting uppercase English to lowercase, converting full-width Chinese characters to half-width Chinese characters, and deleting stop words in the corpus training data.

[0059] To further illustrate, in the process of unifying language standards, some proper names are not converted into uppercase and lowercase English names, for example, Mr., Scottland and other proper nouns remain unchanged.

[0060] It is necessary to further illustrate that, in the process of data cleaning, abbreviated phrases are converted into full phrases, such as converting I'll into Iwill and converting she's into sheis.

[0061] It is necessary to further illustrate that, in the process of data cleaning, words or phrases with the same meaning but different writing styles are converted into preset words or phrases. For example, USA and America both mean the United States, so they are all uniformly converted into America.

[0062] The original corpus data is segmented into sentences and words using natural language processing tools, so that the original corpus data is presented in sentences and word units (tokens) corresponding to the sentences. For example, the sentence "the inventor performs word segmentation on the sentence, and counts the length of the sentence word segmentation list and the word segmentation set" is segmented, and the word units obtained are: inventor, to, sentence, perform, word segmentation, process, and, count, sentence, word segmentation, list, and, word segmentation, set, of, length.

[0063] Step 103: Determine a word unit list and a word unit set according to the plurality of word units, wherein the word unit set is a set after removing repeated word units in the word unit list;

[0064] Step 104: Calculate the word unit list length and word unit set length corresponding to the sentence, wherein the word unit list length is the number of word units in the sentence, and the word unit set length is the number of word units after removing repeated word units in the word unit list;

[0065] Step 105, calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judging whether the sentence meets the preset conditions;

[0066] For example, the sentence "the inventor performs word segmentation processing on the sentence, and counts the length of the word segmentation list and the word segmentation set of the sentence" has a word unit list after word segmentation of (inventor, to, sentence, perform, word segmentation, process, and, count, sentence, word segmentation, list, and, word segmentation, set, of, length), the word unit list length corresponding to the sentence is 16, the word unit set after word segmentation of the sentence is (inventor, to, sentence, perform, word segmentation, process, and, count, list, and, set, of, length), the word unit set length corresponding to the sentence is 13, and accordingly, the ratio of the word unit list length to the word unit set length corresponding to the sentence is 16 / 13. Those skilled in the art can set a preset threshold value according to actual needs, calculate the ratio of the word unit list length and the word unit set length corresponding to the sentence, and when the value of the ratio is less than the predetermined threshold value, it is determined that the sentence meets the preset condition, and when the value of the ratio is greater than or equal to the preset threshold value, it is determined that the sentence does not meet the preset condition.

[0067] Step 106: Filter the sentences that do not meet the preset conditions, and add the sentences that meet the preset conditions to the corpus training database.

[0068] The present invention performs data cleaning on original corpus data to obtain candidate corpus data, performs word segmentation on sentences in the candidate corpus data to obtain a plurality of word units, calculates the ratio of the word unit list length corresponding to the sentence to the word unit set length, determines whether the sentence meets preset conditions, adds the sentences that meet the preset conditions to a corpus training database, and uses them for re-training of a neural network machine translation model, thereby reducing the probability of continuous recurrence of the same token in the sentences obtained after the neural network machine translation, improving the translation experience of the neural network machine, and improving the translation effect of the neural network machine.

[0069] In an optional embodiment, the candidate corpus data is a monolingual sentence, which is a sentence containing one language. The first word unit list length and the first word unit set length of the monolingual sentence are calculated, where the first word unit list length is the number of word units in the monolingual sentence, and the first word unit set length is the number of first word units after removing repeated word units in the first word unit list. The first word segmentation length ratio is calculated, where the first word segmentation length ratio is the ratio of the first word unit list length to the first word unit set length.

[0070] Exemplarily, the list of word units after tokenizing the sentence "The candidate corpus data is a monolingual corpus sentence, and the monolingual corpus sentence is a sentence containing one language" is (candidate, corpus, data, is, monolingual corpus, sentence, monolingual corpus, sentence, is, contains, one, language, of, sentence). It should be particularly noted that due to different natural language tokenization tools or tokenization standards used by the inventors, the obtained tokens (word units) may vary. For example, in this embodiment, "corpus" and "monolingual corpus" are regarded as two different tokens. The length of the first word unit list corresponding to this sentence is 14, and the set of word units after tokenizing this sentence is (candidate, corpus, data, is, monolingual corpus, sentence, contains, one, language, of), and the length of the first word unit set corresponding to this sentence is 10. Correspondingly, the ratio of the length of the first tokenization of this sentence is 1.4. Those skilled in the art can set a preset first threshold according to actual needs, calculate the ratio of the length of the first word unit list and the length of the first word unit set corresponding to the monolingual corpus sentence. When the value of this ratio is less than the preset first threshold, it is determined that this sentence meets the preset conditions. When the value of this ratio is greater than or equal to the preset first threshold, it is determined that this sentence does not meet the preset conditions.

[0071] In an alternative embodiment, the candidate corpus data includes parallel corpus sentences, and the parallel corpus is a sentence written in different languages and having a translation relationship with each other.

[0072] Taking Chinese-English mutual translation as an example (source text: A, B, C, D; translation: a, b, c, d), Figure 2 is a schematic diagram of English to Chinese translation provided by an embodiment of the present invention, Figure 3 is a schematic diagram of Chinese to English translation provided by an embodiment of the present invention. The framework of the neural machine translation model generally selects the Encoder-Decoder framework. The Encoder (encoding module) includes one or several layers of neural networks, and the Decoder (decoding module) also includes one or several layers of neural networks, where <eos>The Encoder is used to extract the information of the sentence input into the encoding module and replace the original sentence with an abstract semantics, while the Decoder (decoding module) is used to convert the abstract semantics into the sentence of the target language, and make the generated sentence perfectly express the meaning of the original sentence and conform to the logical habits of the target language.

[0073] When the parallel corpus contains only two languages, this embodiment regards the two languages ​​as a source language sentence and a target language sentence, calculates a second word unit list length and a second word unit set length of the source language sentence, where the second word unit list length is the number of the word units in the source language sentence, and the second word unit set length is the number of the second word units after removing repeated word units in the second word unit list; and calculates a third word unit list length and a third word unit set length of the target language sentence, where the third word unit list length is the number of the word units in the target language sentence, and the third word unit set length is the number of repeated word units in the third word unit list. The method comprises the following steps: calculating the number of the third word units after the word element; calculating the second word segmentation length ratio, which is the ratio of the second word unit list length to the second word unit set length; judging whether the second word segmentation length ratio is less than a preset second threshold; calculating the third word segmentation length ratio, which is the ratio of the third word unit list length to the third word unit set length; judging whether the third word segmentation length ratio is less than a preset third threshold; calculating the fourth word segmentation length ratio, which is the ratio of the second word unit list length to the third word unit list length; judging whether the fourth word segmentation length ratio is within a preset fourth threshold interval.

[0074] The second word length ratio src_ratio = src_list_len / src_set_len,

[0075] The third word length ratio tgt_ratio = tgt_list_len / tgt_set_len,

[0076] Fourth word length ratio src_tgt_ratio = src_list_len / tgt_list_len

[0077] Among them, src_ratio is the second word segmentation length ratio corresponding to the source language sentence, src_list_len is the second word unit list length corresponding to the source language sentence, src_set_len is the second word unit set length corresponding to the source language sentence, tgt_ratio is the third word segmentation length ratio corresponding to the target language sentence, tgt_list_len is the third word unit list length corresponding to the target language sentence, tgt_set_len is the third word unit set length corresponding to the target language sentence, src_tgt_ratio is the fourth word segmentation length ratio corresponding to the parallel corpus sentence, src_list_len is the second word unit list length corresponding to the source language sentence, and src_set_len is the third word unit list length corresponding to the target language sentence.

[0078] Those skilled in the art may set a preset second threshold value according to actual needs. When the ratio of the second word segmentation length is less than the preset second threshold value, it is determined that the source language sentence meets the preset condition. When the value of the ratio is greater than or equal to the preset second threshold value, it is determined that the source language sentence does not meet the preset condition.

[0079] Those skilled in the art may set a preset third threshold value according to actual needs. When the ratio of the third word segmentation length is less than the preset third threshold value, it is determined that the target language sentence meets the preset condition; when the value of the ratio is greater than or equal to the preset third threshold value, it is determined that the target language sentence does not meet the preset condition.

[0080] Those skilled in the art can set a preset fourth threshold interval according to actual needs. When the fourth word segmentation length ratio is within the preset fourth threshold interval, it is determined that the parallel corpus sentence meets the preset condition. When the value of the ratio is not within the preset fourth threshold interval, it is determined that the parallel corpus sentence does not meet the preset condition. For example, taking a parallel corpus containing Chinese sentences and English sentences as an example, word segmentation operations are performed on the Chinese sentences and English sentences respectively. After statistics, it is found that the length of the word unit list of the Chinese words and sentences is a, and the length of the word unit list of the English sentences is b. The minimum length ratio μ and the maximum length ratio λ are set. When μ

[0081] In an optional embodiment, the above corpus screening method further includes:

[0082] When the candidate corpus data is a parallel corpus, the mutual translation matching rate of the parallel corpus is calculated; and whether the parallel corpus meets a preset condition is determined according to the mutual translation matching rate.

[0083] ​Taking a parallel corpus containing a Chinese sentence and an English sentence as an example, perform word segmentation operations on the Chinese sentence and the English sentence respectively. There are X word units in the segmented Chinese sentence. Extract x (x < X) word units from them. Translate these x word units into the corresponding x English word units through a dictionary or a general translation tool. There are y word units in the segmented English sentence that can exactly match these x English word units. Then the first mutual translation matching rate p1 = x / y. Similarly, there are Y word units in the segmented English sentence. Extract m word units from them. Translate these m word units into the corresponding m Chinese word units through a dictionary or a general translation tool. There are n word units in the segmented Chinese sentence that can exactly match these m English word units. Then the second mutual translation matching rate p2 = m / n. The mutual translation matching rate of the parallel corpus is p = (p1 + p2) / 2. Those skilled in the art can set a preset fifth threshold according to actual needs. When the mutual translation matching rate of the parallel corpus is greater than the preset fifth threshold, it is determined that the parallel corpus meets the preset conditions. Otherwise, the parallel corpus is filtered.

[0084] It should be noted that for the method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0085] To implement the above-mentioned corpus screening method, an embodiment of the present invention further provides a corpus screening device, including:

[0086] An original corpus data acquisition module 201, configured to acquire original corpus data;

[0087] A first word unit acquisition module 202, configured to perform data cleaning on the original corpus data to obtain candidate corpus data, where the candidate corpus data includes multiple sentences, and perform word segmentation on the multiple sentences to obtain a plurality of word units;

[0088] A second word unit acquisition module 203, configured to determine a word unit list and a word unit set according to the plurality of word units, where the word unit set is a set after removing the duplicate word units in the word unit list;

[0089] A third word unit acquisition module 204, configured to calculate the length of the word unit list and the length of the word unit set corresponding to the sentence, where the length of the word unit list is the number of word units in the sentence, and the length of the word unit set is the number of word units after removing the duplicate word units in the word unit list;

[0090] A judgment module 205 is used to calculate the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judge whether the sentence meets the preset conditions;

[0091] The corpus training database acquisition module 206 is used to filter the sentences that do not meet the preset conditions and add the sentences that meet the preset conditions to the corpus training database.

[0092] In an optional embodiment, the candidate corpus data includes a single corpus sentence, and the third word unit acquisition module 204 may include:

[0093] The first word unit acquisition submodule is used to calculate the first word unit list length and the first word unit set length of the monolingual sentence, wherein the first word unit list length is the number of the word units in the monolingual sentence, and the first word unit set length is the number of the first word units after removing repeated word units in the first word unit list.

[0094] In an optional embodiment, the determining module 205 may include:

[0095] A first calculation submodule, used for calculating a first word segmentation length ratio, where the first word segmentation length ratio is a ratio of the first word unit list length to the first word unit set length;

[0096] The first judgment submodule is used to judge whether the first word segmentation length ratio is less than a preset first threshold.

[0097] In an optional embodiment, the candidate corpus data includes parallel corpus sentences, and the parallel corpus sentences include source language sentences and target language sentences. The third word unit acquisition module 204 may include:

[0098] A second word unit acquisition submodule, configured to calculate a second word unit list length and a second word unit set length of the source language sentence, wherein the second word unit list length is the number of the word units in the source language sentence, and the second word unit set length is the number of the second word units after removing repeated word units in the second word unit list;

[0099] The third word unit acquisition submodule is used to calculate the third word unit list length and the third word unit set length of the target language sentence, wherein the third word unit list length is the number of the word units in the target language sentence, and the third word unit set length is the number of the third word units after removing repeated word units in the third word unit list.

[0100] In an optional embodiment, the determining module 205 may include:

[0101] A second calculation submodule, used for calculating a second word segmentation length ratio, where the second word segmentation length ratio is a ratio of the second word unit list length to the second word unit set length;

[0102] A second judgment submodule, used to judge whether the second word segmentation length ratio is less than a preset second threshold;

[0103] A third calculation submodule, used for calculating a third word segmentation length ratio, wherein the third word segmentation length ratio is a ratio of the third word unit list length to the third word unit set length;

[0104] A third judgment submodule, used to judge whether the third word segmentation length ratio is less than a preset third threshold;

[0105] a fourth calculation submodule, configured to calculate a fourth word segmentation length ratio, wherein the fourth word segmentation length ratio is a ratio of the second word unit list length to the third word unit list length;

[0106] The fourth judgment submodule is used to judge whether the fourth word segmentation length ratio is within a preset fourth threshold range.

[0107] Based on the above description, the present invention obtains candidate corpus data by performing data cleaning on the original corpus data, segmenting the sentences in the candidate corpus data to obtain a number of word units, calculating the ratio of the word unit list length and the word unit set length corresponding to the sentence, judging whether the sentence meets the preset conditions, adding the sentences that meet the preset conditions to the corpus training database, and using them for re-training of the neural network machine translation model, thereby reducing the probability of continuous recurrence of the same token in the sentences obtained after the neural network machine translation, improving the translation experience of the neural network machine, and improving the translation effect of the neural network machine.

[0108] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned corpus screening embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned corpus screening method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0110] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0111] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0112] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0116] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0117] The above is a detailed introduction to a corpus screening method and a corpus screening device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.< / eos>

Claims

1. A corpus screening method, characterized in that: include: Get the original corpus data; Performing data cleaning on the original corpus data to obtain candidate corpus data, wherein the candidate corpus data includes a plurality of sentences, and performing word segmentation on the plurality of sentences to obtain a plurality of word units; Determine a word unit list and a word unit set according to the plurality of word units, wherein the word unit set is a set after removing repeated word units in the word unit list; Calculate the word unit list length and word unit set length corresponding to the sentence, wherein the word unit list length is the number of the word units in the sentence, and the word unit set length is the number of the word units after removing repeated word units in the word unit list; Calculate the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judge whether the sentence meets the preset condition. When the value of the ratio is less than a preset threshold, judge that the sentence meets the preset condition; when the value of the ratio is greater than or equal to the preset threshold, judge that the sentence does not meet the preset condition. Filter out sentences that do not meet the preset conditions, and add sentences that meet the preset conditions to the corpus training database.

2. The method according to claim 1, characterized in that The candidate corpus data includes a single corpus sentence, and the step of calculating the word unit list length and the word unit set length of the sentence includes: Calculate the first word unit list length and the first word unit set length of the monolingual sentence, where the first word unit list length is the number of the word units in the monolingual sentence, and the first word unit set length is the number of the first word units after removing repeated word units in the first word unit list.

3. The method according to claim 2, characterized in that Calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judging whether the sentence meets the preset condition comprises: Calculating a first word segmentation length ratio of the single language sentence, where the first word segmentation length ratio is a ratio of the first word unit list length to the first word unit set length; It is determined whether the first word segmentation length ratio is less than a preset first threshold.

4. The method according to claim 1, characterized in that: The candidate corpus data includes parallel corpus sentences, the parallel corpus sentences include source language sentences and target language sentences, and the calculating of the word unit list length and word unit set length corresponding to the sentences includes: Calculating a second word unit list length and a second word unit set length of the source language sentence, wherein the second word unit list length is the number of the word units in the source language sentence, and the second word unit set length is the number of the second word units after removing repeated word units in the second word unit list; The third word unit list length and the third word unit set length of the target language sentence are calculated, wherein the third word unit list length is the number of the word units in the target language sentence, and the third word unit set length is the number of the third word units after removing repeated word units in the third word unit list.

5. The method according to claim 4, characterized in that Calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, and judging whether the sentence meets the preset condition comprises: Calculating a second word segmentation length ratio of the source language sentence, where the second word segmentation length ratio is a ratio of the second word unit list length to the second word unit set length; Calculating a third word segmentation length ratio of the target language sentence, wherein the third word segmentation length ratio is a ratio of the third word unit list length to the third word unit set length; Calculating a fourth word segmentation length ratio of the parallel corpus sentence, wherein the fourth word segmentation length ratio is a ratio of the second word unit list length to the third word unit list length; It is determined whether the second word segmentation length ratio is less than a preset second threshold, whether the third word segmentation length ratio is less than a preset third threshold, and whether the fourth word segmentation length ratio is within a preset fourth threshold interval.

6. A corpus screening device, characterized in that: include: The original corpus data acquisition module is used to acquire the original corpus data; A first word unit acquisition module is used to perform data cleaning on the original corpus data to obtain candidate corpus data, wherein the candidate corpus data includes a plurality of sentences, and segment the plurality of sentences to obtain a plurality of word units; A second word unit acquisition module, configured to determine a word unit list and a word unit set according to the plurality of word units, wherein the word unit set is a set after removing repeated word units in the word unit list; A third word unit acquisition module, used to calculate the word unit list length and word unit set length corresponding to the sentence, wherein the word unit list length is the number of the word units in the sentence, and the word unit set length is the number of the word units after removing repeated word units in the word unit list; A judgment module, used for calculating the ratio of the length of the word unit list corresponding to the sentence to the length of the word unit set, judging whether the sentence meets the preset condition, and judging that the sentence meets the preset condition when the value of the ratio is less than a preset threshold, and judging that the sentence does not meet the preset condition when the value of the ratio is greater than or equal to the preset threshold; The corpus training database acquisition module is used to filter out sentences that do not meet the preset conditions and add sentences that meet the preset conditions to the corpus training database.

7. The device according to claim 6, characterized in that The candidate corpus data includes a single corpus sentence, and the third word unit acquisition module includes: The first word unit acquisition submodule is used to calculate the first word unit list length and the first word unit set length of the monolingual sentence, wherein the first word unit list length is the number of the word units in the monolingual sentence, and the first word unit set length is the number of the first word units after removing repeated word units in the first word unit list.

8. The device according to claim 7, characterized in that The judging module comprises: A first calculation submodule, used for calculating a first word segmentation length ratio, where the first word segmentation length ratio is a ratio of the first word unit list length to the first word unit set length; The first judgment submodule is used to judge whether the first word segmentation length ratio is less than a preset first threshold.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the corpus screening method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the corpus screening method according to any one of claims 1 to 5 are implemented.

Images