Corpus filtering method and device based on IBMmodel

Abstract

The present invention provides a corpus filtering method and device based on IBMmodel, by collecting a large amount of training corpus in a target language, including an original text and its corresponding standard translation; segmenting the original text and its corresponding standard translation to obtain a first set; inputting the first set into an IBMModel model for training to obtain a word alignment probability distribution and a word position alignment probability distribution; segmenting the target corpus to be filtered to obtain a second set; calculating the alignment probability score between language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution; judging whether to filter the corpus by the alignment probability score. Using this method, it is possible to judge whether the semantics between language pairs are consistent, thereby achieving the effect of efficiently filtering dirty data.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a corpus filtering method, apparatus, computer equipment and storage medium based on IBM model. Background Art

[0002] Corpus filtering methods have always been an important task in the field of natural language processing. Compared with the existing parallel corpora of each language pair translated by human translators, the number of corpora crawled from the Internet or automatically generated by machine translation is much larger than the former, but the quality of the latter is not satisfactory. Since NMT (Neural Machine Translation) is more sensitive to noise than SMT (Statistical Machine Translation), the corpora crawled from the Internet or automatically generated by machine translation often cannot be directly used as training corpus data for NMT. ​​Therefore, especially for low-resource language pair corpora, whether it can extract usable parallel language pairs from these large amounts of dirty data is particularly important for NMT.

[0003] Existing corpus filtering technologies are basically based on various fixed rules provided by human linguistics experts to determine whether the language pair meets the standards as the basis for implementing corpus filtering. Rules can indeed filter out some obvious dirty data, but this type of method cannot identify the semantic similarity of parallel corpora, that is, it cannot filter out semantically inconsistent corpora (for example, zh: Today's weather is very good. en: You are a nice guy.), and most corpora generated by web crawling or machine translation contain a large amount of semantically inconsistent dirty data.

[0004] Therefore, there is an urgent need for a corpus filtering method that can identify the semantic similarity of parallel corpora to solve the problem that existing corpus filtering technology cannot filter semantically inconsistent corpora, thereby achieving the effect of efficiently filtering dirty data. Summary of the invention

[0005] The embodiment of the present invention provides a corpus filtering method, apparatus, computer equipment and storage medium based on IBM model, so as to solve the problem that the existing corpus filtering technology cannot filter corpus with inconsistent semantics.

[0006] In order to achieve the above object, a first aspect of an embodiment of the present invention provides a corpus filtering method based on IBM model, comprising:

[0007] Collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation;

[0008] Segment the original text and its corresponding standard translation to obtain a first set;

[0009] Input the first set into the IBM Model for training to obtain a word alignment probability distribution and a word position alignment probability distribution;

[0010] Perform word segmentation on the target corpus to be filtered to obtain a second set;

[0011] Calculating alignment probability scores between language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution;

[0012] Whether to filter the corpus is determined based on the alignment probability score.

[0013] Optionally, in a possible implementation manner of the first aspect, inputting the first set into an IBM Model model for training to obtain a word alignment probability distribution and a word position alignment probability distribution includes:

[0014] Step 1: Initialize the word alignment probability distribution and word position alignment probability distribution of all language pairs in the first set to obtain the word alignment initial value and word position alignment initial value;

[0015] Step 2: Calculate the sum of the word alignment probability distributions based on the word alignment initial values ​​of all language pairs, and calculate the sum of the word position alignment probability distributions based on the word position alignment initial values ​​of all language pairs;

[0016] Step 3: Based on the sum of the word alignment probability distribution and the sum of the word position alignment distribution, recalculate the word alignment probability distribution and the word position alignment probability distribution;

[0017] Step 4: Repeat steps 2 and 3 until the word alignment probability distribution and the word position alignment probability distribution no longer change.

[0018] Optionally, in a possible implementation manner of the first aspect, calculating the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution includes:

[0019] Calculating a first alignment function by using the word alignment probability distribution and the word position alignment probability distribution;

[0020] a=argmax a P(a|E,F)

[0021] Among them, a represents the alignment function, E and F represent the word vectors of the original and translated texts after word segmentation, respectively. E = [e1, e2, e3, ..., em] T ,F=[f1,f2,f3,……,fn]T , argmax represents finding the parameter with the maximum score, argmax a P(aE,F) represents finding the alignment function that maximizes the P(aE,F) value, and P(a|E,F) represents the product of the word alignment probability distribution of the original text and the word position alignment probability distribution.

[0022] Optionally, in a possible implementation manner of the first aspect, calculating the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution includes:

[0023] Reverse the positions of the original text and the translation in the first set;

[0024] Input the inverted first set into the IBM Model model to obtain a new word alignment probability distribution and word position alignment probability distribution;

[0025] Based on the new word alignment probability distribution and the word position alignment probability distribution, a second alignment function is calculated.

[0026] Optionally, in a possible implementation manner of the first aspect, calculating the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution includes:

[0027] Calculating alignment probability scores between language pairs in the second set based on the first alignment function and the second alignment function;

[0028]

[0029] Among them, score represents the alignment probability score, n represents the number of translated words, m represents the number of original words, P(e a(i) |f i ) represents the i-th word in the translation and the corresponding word in the original text in the second set a(i) The word alignment probability, P(f ra(j) |e j ) represents the jth word in the translation and the corresponding word in the original text in the second set after the position is inverted ra(j) The word alignment probability of , P(a(i)|i, m, n) represents the position alignment probability of the i-th position of the translation in the second set being aligned with the a(i)-th position of the original text, and P(ra(j)|j, m, n) represents the position alignment probability of the j-th position of the translation in the second set after position inversion being aligned with the ra(j)-th position of the original text.

[0030] Optionally, in a possible implementation manner of the first aspect, determining whether to filter the corpus according to the alignment probability score includes:

[0031] If the alignment probability score is greater than or equal to a preset threshold, it is determined that the corpus does not need to be filtered;

[0032] If the alignment probability score is less than a preset threshold, it is determined to be a corpus that needs to be filtered.

[0033] A second aspect of an embodiment of the present invention provides a corpus filtering device based on IBM model, comprising:

[0034] The collection module is used to collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation;

[0035] A first word segmentation module is used to segment the original text and its corresponding standard translation to obtain a first set;

[0036] A probability distribution calculation module, used for inputting the first set into an IBM Model for training to obtain a word alignment probability distribution and a word position alignment probability distribution;

[0037] The second word segmentation module is used to perform word segmentation processing on the target corpus to be filtered to obtain a second set;

[0038] A probability score calculation module, configured to calculate the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution;

[0039] A judgment module is used to judge whether to filter the corpus based on the alignment probability score.

[0040] Optionally, in a possible implementation manner of the second aspect, the probability distribution calculation module includes:

[0041] A probability distribution initialization unit, used to initialize the word alignment probability distribution and the word position alignment probability distribution of all language pairs in the first set to obtain the word alignment initial value and the word position alignment initial value;

[0042] A first calculation unit, configured to calculate a sum of word alignment probability distributions according to word alignment initial values ​​of all language pairs, and to calculate a sum of word position alignment probability distributions according to word position alignment initial values ​​of all language pairs;

[0043] A second calculation unit, configured to recalculate the word alignment probability distribution and the word position alignment probability distribution based on the probability sum of the word alignment probability distribution and the probability sum of the word position alignment distribution;

[0044] The repetitive execution unit is used to repeatedly execute the first calculation unit and the second calculation unit until the word alignment probability distribution and the word position alignment probability distribution no longer change.

[0045] According to a third aspect of an embodiment of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0046] According to a fourth aspect of an embodiment of the present invention, a readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the steps of the method of the first aspect of the present invention and various possible designs of the first aspect.

[0047] The corpus filtering method, device, computer equipment and storage medium based on IBM model provided by the present invention collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation; segment the original text and its corresponding standard translation to obtain a first set; input the first set into the IBM Model model for training to obtain the word alignment probability distribution and the word position alignment probability distribution; segment the target corpus to be filtered to obtain a second set; calculate the alignment probability score between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution; and determine whether to filter the corpus by the alignment probability score. The present invention introduces the IBM Model model to find the probability distribution of the alignment of each word between the language pairs and the probability distribution of the word position alignment between the language pairs in the parallel corpus. When it is necessary to determine whether the new corpus is a translation language pair, it can be determined whether it is an available parallel corpus by the total probability of the alignment of each word. Using this method, it is possible to determine whether the semantics between the language pairs are consistent, thereby achieving the effect of efficiently filtering dirty data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of the first implementation method of the corpus filtering method based on IBMmodel;

[0049] Figure 2 It is a schematic diagram of one-to-many mode;

[0050] Figure 3 This is a structural diagram of the first implementation of a corpus filtering device based on IBM model. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0053] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0054] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0055] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0056] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0057] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0058] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] The present invention provides a corpus filtering method based on IBM model, such as Figure 1 The flow chart shown includes:

[0060] Step S110: Collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation.

[0061] In this step, a massive parallel corpus will be searched for the target language of the translation, which will be used as the training corpus in the subsequent IBM model training. The massive parallel corpus refers to the existing high-quality corpus, including the original translation and the corresponding standard translation.

[0062] Step S120: Segment the original text and its corresponding standard translation to obtain a first set.

[0063] In step S120, after obtaining the training corpus of the target language, it is necessary to perform word segmentation processing on it. Among them, word segmentation processing refers to dividing the entire sentence into unit words, including: characters, words, punctuation numbers, etc. For example: The weather is very good today. --(word segmentation processing)→[Today, the weather is very good.]. It is necessary to perform word segmentation processing on both the original text and the translation in the training corpus. For Chinese, jieba can be used for word segmentation processing, and English or other languages ​​can use word segmentation tools such as nltk or spacy. Since the effect of word segmentation will affect the effect of the final word alignment, the present invention uses jieba for Chinese, and uses spacy, which has a more ideal effect, for other languages. For word segmentation processing. On the contrary, if the training corpus is not subjected to word segmentation processing, the subsequent IBM model model will not be able to learn the relationship between words.

[0064] Step S130: input the first set into the IBM Model for training to obtain a word alignment probability distribution and a word position alignment probability distribution.

[0065] In this step, there are 5 IBM Model models, namely IBM model 1, 2, 3, 4, and 5. The IBM model 2 model is selected in the present invention, and it can also be replaced by the other four models. The trained IBM model 2 model mainly searches for word alignment probability distribution and word position alignment probability distribution in the parallel corpus after word segmentation through the EM algorithm. Wherein the word alignment probability distribution is P (ef), e and f are two words in the original text and the translated text, and P (ef) is the probability that the word in the original text corresponds to the word in the translated text. The word position alignment distribution is P (ij, m, n), m, n are the lengths of the original text and the translated text respectively, j is the position of a certain word in the original text, and i is the position of a certain word in the translated text, that is, P (ij, m, n) is: on the basis of the original text and the translated text length being m and n, the word with word position j in the original text corresponds to the word with word position i in the translated text.

[0066] In addition, in step S130, the training steps of the IBM model 2 model are as follows:

[0067] Step 1: Initialize the word alignment probability distribution P(ef) and word position alignment probability distribution P(ij,m,n) of all language pairs in the first set to obtain the word alignment initial value and the word position alignment initial value;

[0068] Step 2: Calculate the sum of word alignment probability distribution c(ef) according to the initial word alignment values ​​of all language pairs, and calculate the sum of word position alignment probability distribution c(ij,m,n) according to the initial word position alignment values ​​of all language pairs, that is, (calculate the sum of the probability of each alignment);

[0069] Step 3: Based on the sum of word alignment probability distribution c(ef) and the sum of word position alignment distribution c(ij,m,n), recalculate the word alignment probability distribution P(ef) and the word position alignment probability distribution P(ij,m,n);

[0070] Step 4: Repeat steps 2 and 3 until the word alignment probability distribution P(ef) and the word position alignment probability distribution P(ij,m,n) no longer change.

[0071] Among them, step 2 is called the E step, step 3 is called the M step, and finally the converged parameters P(ef) and P(ij,m,n) are stored in the json file.

[0072] Step S140: perform word segmentation processing on the target corpus to be filtered to obtain a second set.

[0073] In step S140, it is also necessary to perform word segmentation processing on the target filtering corpus, and the specific word segmentation processing steps and the tools used for word segmentation are the same as those in step S120.

[0074] Step S150: Calculate the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution.

[0075] In this step, before calculating the alignment probability score, it is necessary to first search for the word alignment probability distribution P(ef) and the word position alignment probability distribution P(ij,m,n) in the parallel corpus of the first set according to the trained IBM model 2 model, and calculate the first alignment function aliment according to these two probability distributions; then invert the positions of the original text and the translation in the first set, input the inverted first set into the IBM Model model, obtain new word alignment probability distribution and word position alignment probability distribution, and calculate the second alignment function based on the new word alignment probability distribution and word position alignment probability distribution, so as to calculate the alignment probability score according to the first and second alignment functions.

[0076] The first set is a set of massive high-quality training corpora in the target language after word segmentation, including the original text and its corresponding standard translation. Specifically, the target language refers to the same language as the target corpus that needs to be filtered. The second set is a set of target corpora that needs to be filtered after word segmentation. "Reversing the positions of the original text and the translation in the first set" means: Since the IBM model 2 model adopts a one-to-many mode, that is, the words in the original text can be aligned to multiple words in the translation, but not vice versa. Specifically, Figure 2 As shown; therefore, it is necessary to flip the original text and the translation in the language pair of the target corpus to be filtered, that is, the original text becomes the translation, and the translation becomes the original text. Then the flipped corpus is processed by word segmentation and input into the trained IBMmodel 2 model step, and the new P(fe) and P(ji, n, m) are output and stored in the json file. And the second alignment function reversed aliment is obtained based on the new P(fe) and P(ji, n, m). Therefore, the alignment probability score score is calculated based on the first alignment function aliment and the second alignment function reversed aliment.

[0077] Step S160: Determine whether to filter the corpus based on the alignment probability score.

[0078] In this step, it is necessary to set a threshold in advance according to the actual situation, so that when the alignment probability score is greater than or equal to the preset threshold, it is determined as a corpus that does not need to be filtered, that is, it is determined as a legal translation pair with similar semantics; conversely, when the probability score is less than the preset threshold, it is determined as a corpus that needs to be filtered, that is, it is determined as an illegal translation pair with dissimilar semantics, thereby achieving the cleaning of dirty data.

[0079] In one embodiment, the first alignment function is calculated according to the word alignment probability distribution and the word position alignment probability distribution by the following formula:

[0080] a=argmax a P(a|E,F)

[0081] Among them, a represents the alignment function, E and F represent the word vectors of the original and translated text after word segmentation E = [e1, e2, e3, ..., em]T, F = [f1, f2, f3, ..., fn]T respectively, argmax represents finding the parameter with the maximum score, argmaxaP(aE, F) represents finding the alignment function that maximizes the value of P(aE, F), P(a|E, F) represents the product of the word alignment probability distribution of the original and translated texts and the word position alignment probability distribution, the alignment function element a(i) = j, indicating that the i-th word in F, i.e. fi, is aligned with the j-th word in E, i.e. ej. For example: the alignment function a = {1:1, 2:2, 3:3, 4:4, 5:4}, i.e. a(5) = 4, indicating that the 5th word in F is aligned with the 4th word in E.

[0082] In one embodiment, according to the first alignment function and the second alignment function, the following formula is used:

[0083] calculating alignment probability scores between language pairs in the second set;

[0084]

[0085] Among them, score represents the alignment probability score, n represents the number of translated words, m represents the number of original words, P(e a(i) |f i ) represents the i-th word in the translation and the corresponding word in the original text in the second set a(i) The word alignment probability, P(f ra(j) |e j ) represents the jth word in the translation and the corresponding word in the original text in the second set after the position is inverted ra(j) The word alignment probability of , P(a(i)|i, m, n) represents the position alignment probability of the i-th position of the translation in the second set being aligned with the a(i)-th position of the original text, and P(ra(j)|j, m, n) represents the position alignment probability of the j-th position of the translation in the second set after position inversion being aligned with the ra(j)-th position of the original text.

[0086] The corpus filtering method based on IBMmodel provided by the present invention collects a large amount of training corpus in the target language, including the original text and its corresponding standard translation; performs word segmentation on the original text and its corresponding standard translation to obtain a first set; inputs the first set into the IBM Model model for training to obtain the word alignment probability distribution and the word position alignment probability distribution; performs word segmentation on the target corpus to be filtered to obtain a second set; calculates the alignment probability score between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution; and determines whether to filter the corpus by the alignment probability score. The present invention introduces the IBM Model model to find the probability distribution of the alignment of each word between the language pairs and the probability distribution of the word position alignment between the language pairs in the parallel corpus. When it is necessary to determine whether the new corpus is a translation language pair, it can be determined whether it is an available parallel corpus by the total probability of the alignment of each word. Using this method, it is possible to determine whether the semantics between the language pairs are consistent, thereby achieving the effect of efficiently filtering dirty data.

[0087] Technical effects:

[0088] (1) The present invention uses a statistical machine learning algorithm, which enables the filtering method to have a certain ability to identify semantic similarity, breaking away from the limitation of traditional filtering that can only be performed by rules, and greatly enhancing the effectiveness of corpus filtering.

[0089] (2) The present invention has the universality of filtering corpora of various languages. Compared with rule-based filtering, it omits the work of manually designing filtering rules based on language characteristics.

[0090] (3) The present invention can identify the semantic similarity between language pairs to a certain extent, and can filter out most of the corpus with different semantics but which cannot be filtered by traditional rule filtering, thereby greatly improving the corpus quality of the filtered corpus. The present invention can be used to filter web crawler corpus and corpus automatically generated by machine translation, so that the corpus can be used as training corpus for NMT.

[0091] The embodiment of the present invention also provides a corpus filtering device based on IBM model, such as Figure 3 As shown, including:

[0092] The collection module is used to collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation;

[0093] A first word segmentation module is used to segment the original text and its corresponding standard translation to obtain a first set;

[0094] A probability distribution calculation module, used for inputting the first set into an IBM Model for training to obtain a word alignment probability distribution and a word position alignment probability distribution;

[0095] The second word segmentation module is used to perform word segmentation processing on the target corpus to be filtered to obtain a second set;

[0096] A probability score calculation module, configured to calculate the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution;

[0097] A judgment module is used to judge whether to filter the corpus based on the alignment probability score.

[0098] In one embodiment, the probability distribution calculation module includes:

[0099] A probability distribution initialization unit, used to initialize the word alignment probability distribution and the word position alignment probability distribution of all language pairs in the first set to obtain a word alignment initial value and a word position alignment initial value;

[0100] A first calculation unit, configured to calculate a sum of word alignment probability distributions according to word alignment initial values ​​of all language pairs, and to calculate a sum of word position alignment probability distributions according to word position alignment initial values ​​of all language pairs;

[0101] A second calculation unit, configured to recalculate the word alignment probability distribution and the word position alignment probability distribution based on the probability sum of the word alignment probability distribution and the probability sum of the word position alignment distribution;

[0102] The repetitive execution unit is used to repeatedly execute the first calculation unit and the second calculation unit until the word alignment probability distribution and the word position alignment probability distribution no longer change.

[0103] The corpus filtering device based on IBMmodel provided by the present invention collects a large amount of training corpus in the target language, including the original text and its corresponding standard translation; performs word segmentation on the original text and its corresponding standard translation to obtain a first set; inputs the first set into the IBM Model model for training to obtain the word alignment probability distribution and the word position alignment probability distribution; performs word segmentation on the target corpus to be filtered to obtain a second set; calculates the alignment probability score between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution; and determines whether to filter the corpus by the alignment probability score. The present invention introduces the IBM Model model to find the probability distribution of the alignment of each word between the language pairs and the probability distribution of the word position alignment between the language pairs in the parallel corpus. When it is necessary to determine whether the new corpus is a translation language pair, it can be determined whether it is an available parallel corpus by the total probability of the alignment of each word. Using this method, it is possible to determine whether the semantics between the language pairs are consistent, thereby achieving the effect of efficiently filtering dirty data.

[0104] Among them, the readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0105] The present invention also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of a device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the device implements the methods provided in the above various embodiments.

[0106] In the above-mentioned terminal or server embodiment, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A corpus filtering method based on IBMmodel, characterized in that: include: Collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation; Segment the original text and its corresponding standard translation to obtain a first set; Input the first set into the IBMmodel2 model for training to obtain a word alignment probability distribution and a word position alignment probability distribution; Perform word segmentation on the target corpus to be filtered to obtain a second set; Calculating alignment probability scores between language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution; Determining whether to filter the corpus according to the alignment probability score; The calculating the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution comprises: Calculating a first alignment function by using the word alignment probability distribution and the word position alignment probability distribution; Reverse the positions of the original text and the translation in the first set; Input the inverted first set into the trained IBMmodel2 model to obtain a new word alignment probability distribution and word position alignment probability distribution; Calculating a second alignment function based on the new word alignment probability distribution and the word position alignment probability distribution; Calculating alignment probability scores between language pairs in the second set based on the first alignment function and the second alignment function; Among them, score represents the alignment probability score, n represents the number of translated words, m represents the number of original words, P(e a(i) |f i ) represents the i-th word in the translation and the corresponding word in the original text in the second set a(i) The word alignment probability, P(f ra(j) |e j ) represents the jth word in the translation and the corresponding word in the original text in the second set after the position is inverted ra(j) The word alignment probability of , P(a(i)|i,m,n) represents the position alignment probability of the i-th position of the translation in the second set aligned with the a(i)-th position of the original text, and P(ra(j)|j,m,n) represents the position alignment probability of the j-th position of the translation in the second set after position inversion aligned with the ra(j)-th position of the original text.

2. The corpus filtering method based on IBM model according to claim 1 is characterized in that: Determining whether to filter the corpus according to the alignment probability score includes: If the alignment probability score is greater than or equal to a preset threshold, it is determined that the corpus does not need to be filtered; If the alignment probability score is less than a preset threshold, it is determined to be a corpus that needs to be filtered.

3. A corpus filtering device based on IBMmodel, characterized in that: include: The collection module is used to collect a large amount of training corpus in the target language, including the original text and its corresponding standard translation; A first word segmentation module is used to segment the original text and its corresponding standard translation to obtain a first set; a probability distribution calculation module is used to input the first set into the IBMmodel2 model for training to obtain a word alignment probability distribution and a word position alignment probability distribution; The second word segmentation module is used to perform word segmentation processing on the target corpus to be filtered to obtain a second set; A probability score calculation module, configured to calculate the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution; A judgment module, used for judging whether to filter the corpus according to the alignment probability score; The calculating the alignment probability scores between the language pairs in the second set according to the word alignment probability distribution and the word position alignment probability distribution comprises: Calculating a first alignment function by using the word alignment probability distribution and the word position alignment probability distribution; Reverse the positions of the original text and the translation in the first set; Input the inverted first set into the trained IBMmodel2 model to obtain a new word alignment probability distribution and word position alignment probability distribution; Calculating a second alignment function based on the new word alignment probability distribution and the word position alignment probability distribution; Calculating alignment probability scores between language pairs in the second set based on the first alignment function and the second alignment function; Among them, score represents the alignment probability score, n represents the number of translated words, m represents the number of original words, P(e a(i) |f i ) represents the i-th word in the translation and the corresponding word in the original text in the second set a(i) The word alignment probability, P(f ra(j) |e j ) represents the jth word in the translation and the corresponding word in the original text in the second set after the position is inverted ra(j) The word alignment probability of , P(a(i)|i,m,n) represents the position alignment probability of the i-th position of the translation in the second set aligned with the a(i)-th position of the original text, and P(ra(j)|j,m,n) represents the position alignment probability of the j-th position of the translation in the second set after position inversion aligned with the ra(j)-th position of the original text.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images