Corpus alignment method, related device and computer program product

Abstract

The present disclosure provides a corpus alignment method, device, electronic device, computer-readable storage medium and computer program product, which relate to the fields of artificial intelligence technology such as machine translation, natural language processing, and deep learning. A specific implementation of the method includes: after obtaining an initial corpus pair consisting of a first corpus and a second corpus of different languages, determining the initial corpus pair whose forward conditional probability and reverse conditional probability are both greater than a first threshold as a target corpus pair, then using a preset translation model to determine the semantic matching probability of the target corpus pair, and finally, performing corpus alignment processing on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than a second threshold. This implementation can reduce the probability threshold requirement when screening matching pairs based on statistical probability, so that under the premise of ensuring the accuracy of the semantic correspondence between the first corpus and the second corpus, more word pairs that are not significant at the statistical level can be recalled, thereby improving the alignment quality of the corpus.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technologies such as machine translation, natural language processing, and deep learning, and in particular to a corpus alignment method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In translation systems, bilingual dictionaries provide word-level mappings between two languages, which is very helpful for users to quickly learn other languages. Generally, building a bilingual dictionary requires a large number of language experts to align corpora in different languages, which is a long and complex task. Summary of the Invention

[0003] The embodiments of the present disclosure provide a corpus alignment method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0004] In a first aspect, an embodiment of the present disclosure proposes a corpus alignment method, comprising: obtaining an initial corpus pair consisting of a first corpus and a second corpus of different languages; determining a statistically positive conditional probability and a negative conditional probability of the initial corpus pair, and determining the initial corpus pair whose positive conditional probability and negative conditional probability are both greater than a first threshold as a target corpus pair; determining a semantic matching probability of the target corpus pair using a preset translation model; and performing corpus alignment processing on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than a second threshold.

[0005] In a second aspect, an embodiment of the present disclosure proposes a corpus alignment device, comprising: an initial corpus pair acquisition unit, configured to acquire an initial corpus pair consisting of a first corpus and a second corpus of different languages; a target corpus pair determination unit, configured to determine the statistical positive conditional probability and negative conditional probability of the initial corpus pair, and determine the initial corpus pair whose positive conditional probability and negative conditional probability are both greater than a first threshold as a target corpus pair; a semantic matching probability calculation unit, configured to determine the semantic matching probability of the target corpus pair using a preset translation model; and a corpus alignment processing unit, configured to perform corpus alignment processing on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than a second threshold.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the corpus alignment method described in any implementation method of the first aspect when executing the instructions.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the corpus alignment method described in any implementation manner in the first aspect when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the corpus alignment method as described in any implementation manner in the first aspect.

[0009] The corpus alignment method, apparatus, electronic device, computer-readable storage medium, and computer program product provided by the embodiments of the present disclosure obtain an initial corpus pair consisting of a first corpus and a second corpus in different languages, determine the statistical positive conditional probability and negative conditional probability of the initial corpus pair, and determine the initial corpus pair whose positive conditional probability and negative conditional probability are both greater than a first threshold as a target corpus pair. Then, a preset translation model is used to determine the semantic matching probability of the target corpus pair. Finally, corpus alignment processing is performed on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than a second threshold.

[0010] The present disclosure introduces a translation model to perform secondary screening of word pairs in the target corpus determined based on the positive conditional probability and the reverse conditional probability in statistics, so as to reduce the probability threshold requirement when screening the target corpus pairs based on the positive conditional probability and the reverse conditional probability. Under the premise of ensuring the accuracy of the semantic correspondence between the first corpus and the second corpus, more corpus pairs that are not significant at the statistical level can be recalled, thereby improving the alignment quality of the corpus.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flowchart of a corpus alignment method provided in an embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a method for determining semantic matching probability provided by an embodiment of the present disclosure;

[0016] Figure 4A flowchart of another method for determining semantic matching probability provided by an embodiment of the present disclosure;

[0017] Figure 5 A flowchart of a corpus alignment method in an application scenario provided by an embodiment of the present disclosure;

[0018] Figure 6 A structural block diagram of a corpus alignment device provided in an embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram of the structure of an electronic device suitable for executing a corpus alignment method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0021] In addition, in the technical solutions involved in this disclosure, the acquisition, storage, use, processing, transportation, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the corpus alignment method, apparatus, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0023] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0024] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed, such as corpus alignment applications, corpus matching applications, and instant messaging applications.

[0025] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0026] The server 105 can provide various services through various built-in applications. For example, a corpus alignment application that can query whether corpora in different languages ​​correspond to each other can be provided. When the server 105 runs the corpus alignment application, it can achieve the following effects: first, an initial corpus pair consisting of a first corpus and a second corpus in different languages ​​is obtained from the terminal devices 101, 102, and 103 through the network 104; then, the server 105 determines the statistical positive conditional probability and negative conditional probability of the initial corpus pair, and determines the initial corpus pair whose positive conditional probability and negative conditional probability are both greater than a first threshold as a target corpus pair; next, the server 105 uses a preset translation model to determine the semantic matching probability of the target corpus pair; finally, the server 105 performs corpus alignment processing on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than a second threshold.

[0027] It should be noted that, in addition to being obtained from the terminal devices 101, 102, and 103 via the network 104, the initial corpus pair consisting of the first and second corpora in different languages ​​can also be pre-stored locally on the server 105 in various ways. Therefore, when the server 105 detects that such data is already stored locally (for example, when it begins processing a previously saved corpus alignment task), it can choose to directly obtain such data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0028] Since using a translation model to determine the semantic matching probability of a target corpus pair requires a relatively large amount of computing resources and a relatively strong computing power, the corpus alignment methods provided in the subsequent embodiments of this disclosure are generally performed by a server 105 having relatively strong computing power and a relatively large number of computing resources. Accordingly, the corpus alignment apparatus is generally also disposed in the server 105. However, it should also be noted that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also use the corpus alignment application installed thereon to complete the various operations previously assigned to the server 105, thereby outputting the same results as the server 105. In particular, when there are multiple terminal devices with different computing powers at the same time, but the corpus alignment application determines that the terminal device where it is located has relatively strong computing power and a relatively large amount of remaining computing resources, the terminal device can be allowed to perform the aforementioned operations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the corpus alignment apparatus can also be disposed in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0029] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0030] Please refer to Figure 2 , Figure 2 This is a flowchart of a corpus alignment method provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0031] Step 201: Obtain an initial corpus pair consisting of a first corpus and a second corpus in different languages.

[0032] In this embodiment, the execution subject of the corpus alignment method (eg Figure 1 The server 105 shown obtains an initial corpus pair consisting of a first corpus and a second corpus of different languages. For example, when the language of the first corpus is Chinese, the language of the second corpus can be English, Japanese, German, French, etc., wherein, according to the specific application scenario, the composition of the initial corpus pair can be adjusted to a specific one of the composition modes such as only one first corpus and one second corpus, one first corpus and multiple second corpora, multiple first corpora and one second corpora, and multiple first corpora and multiple second corpora, so as to achieve one-to-many or many-to-many corpus alignment.

[0033] It should be noted that the initial corpus pair consisting of the first corpus and the second corpus in different languages ​​can be directly obtained by the above-mentioned execution subject from a local storage device, or from a non-local storage device (such as Figure 1 The local storage device may be a data storage module provided in the execution subject, such as a server hard disk. In this case, the initial corpus pair consisting of the first corpus and the second corpus in different languages ​​can be quickly read locally. The non-local storage device may also be any other electronic device configured to store data, such as some user terminals. In this case, the execution subject may obtain the required initial corpus pair consisting of the first corpus and the second corpus in different languages ​​by sending a retrieval command to the electronic device.

[0034] Step 202 : Determine the forward conditional probability and the reverse conditional probability of the initial corpus pair, and determine the initial corpus pair whose forward conditional probability and reverse conditional probability are both greater than a first threshold as the target corpus pair.

[0035] In this embodiment, after an initial corpus pair consisting of a first corpus and a second corpus of different languages ​​is obtained based on step 201, a statistically positive conditional probability and a statistically negative conditional probability between the first corpus and the second corpus in the initial corpus pair are determined. The statistically positive conditional probability and the statistically negative conditional probability are obtained by counting the frequencies of positive (from the first corpus to the second corpus) and negative (from the second corpus to the first corpus) correspondences between the first corpus and the second corpus in large-scale bilingual parallel sentences (the languages ​​corresponding to the first corpus and the second corpus). The positive conditional probability and the statistically negative conditional probability between the first corpus and the second corpus in the initial corpus pair can generally be determined using a statistical machine translation (SMT) approach, such as a Moses translation model or a hidden Markov model (HMM).

[0036] Furthermore, the first threshold is usually pre-configured with the ratio of the target corpus that is expected to complete the corpus alignment process to the initial corpus pair. After obtaining the statistical positive conditional probability and negative conditional probability of the initial corpus pair, the numerical relationship between the positive conditional probability and the negative conditional probability and the first threshold is judged, wherein the first threshold is usually set corresponding to the positive conditional probability and the negative conditional probability. When both the positive conditional probability and the negative condition meet the requirements of the first threshold (greater than the first threshold), the initial corpus pair is determined as the target corpus pair.

[0037] In practice, if the forward conditional probability and the reverse conditional probability are determined as a single forward conditional probability and reverse conditional probability by calculating the average value (for example, the first corpus and / or the second corpus is a set consisting of multiple corpora), the first threshold can also be a separate value to facilitate screening of the single forward conditional probability and reverse conditional probability.

[0038] In addition, the forward conditional probability and reverse conditional probability between each first corpus and each second corpus can be generated in advance in the above-mentioned manner, and then a corresponding conditional probability table and conditional probability database can be generated, so that the statistical forward conditional probability and reverse conditional probability of the initial corpus pair can be determined directly based on the conditional probability table and conditional probability database, so that the conditional probability table and conditional probability database can be used for reuse, thereby improving the efficiency of determining the statistical forward conditional probability and reverse conditional probability of the initial corpus pair, and when a corpus similar to the first corpus and the second corpus stored in the conditional probability table and conditional probability database is obtained, the conditional probability table and conditional probability database can be used to determine the forward conditional probability and reverse conditional probability of the initial corpus pair, thereby improving efficiency.

[0039] Step 203: Determine the semantic matching probability of the target corpus pair using a preset translation model.

[0040] In this embodiment, a preset translation model is used to determine the probability that the result obtained by translating the first corpus in the target corpus pair into the language of the second corpus is the second corpus, and the probability that the result obtained by translating the second corpus in the target corpus pair into the language of the first corpus is the first corpus, and then the above two probabilities are summarized and determined.

[0041] Among them, the training method of the translation model from the language of the first corpus to the language of the second corpus can be pre-trained with a first sample in the language of the first corpus and a second sample in the language of the second corpus as input, and the similarity between the content obtained by translating the first sample into the language of the second sample and the second sample as output. Similarly, the training method from the language of the second corpus to the language of the first corpus is similar to the above method. It is only necessary to adjust the output of the translation model during training, and no further details will be given here.

[0042] Furthermore, the translation model can simultaneously realize a single or bilingual translation model that translates the language of the first corpus into the language of the second corpus and the language of the second corpus into the language of the first corpus. It can also be a single translation model composed of two translation models, and the same effect as the single or bilingual translation model can be obtained by combining the two translation models.

[0043] Step 204 : performing corpus alignment processing on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than the second threshold.

[0044] In this embodiment, when the semantic matching probability determined in step 203 is greater than a pre-configured second threshold, the first corpus and the second corpus included in the target corpus pair are aligned, that is, the first corpus and the second corpus are determined to be corpora with different languages ​​but the same semantics.

[0045] The corpus alignment method provided by the embodiments of the present disclosure introduces a translation model to perform secondary screening of word pairs in the target corpus determined based on positive conditional probability and reverse conditional probability, so as to reduce the probability threshold requirement when screening target corpus pairs based on positive conditional probability and reverse conditional probability. Under the premise of ensuring the accuracy of the semantic correspondence between the first corpus and the second corpus, more corpus pairs that are not significant at the statistical level can be recalled, thereby improving the alignment quality of the corpus.

[0046] In some optional implementations of this embodiment, the corpus alignment method further includes: obtaining an initial first corpus set and an initial second corpus set; performing data cleaning on the corpus in the initial first corpus set and the initial second corpus set, respectively, to obtain a cleaned first corpus set and a cleaned second corpus set; performing word segmentation processing on the cleaned first corpus set and the cleaned second corpus set, respectively, to obtain a processed first corpus set and a processed second corpus set; and extracting the first corpus and the second corpus from the processed first corpus set and the processed second corpus set, respectively.

[0047] Specifically, an initial first corpus and an initial second corpus are obtained, wherein the initial first corpus includes at least one first corpus and the initial second corpus includes at least one second corpus. After obtaining the initial first corpus and the initial second corpus, data cleaning is performed on the initial first corpus and the initial second corpus respectively. Data cleaning mainly includes removing redundant spaces, removing garbled characters, normalizing punctuation and other operations. After completing data cleaning and obtaining the processed first corpus and the processed second corpus, word segmentation is performed on the first corpus and the processed second corpus respectively. In the word segmentation processing, the processed first corpus and the processed second corpus obtained after data cleaning will be semantically segmented according to the language usage rules of the language to which the first corpus belongs and the language to which the second corpus belongs, to obtain at least one first corpus and second corpus that meet the requirements of fine-grainedness, so as to pre-process the first corpus and the second corpus by means of data cleaning and word segmentation to improve the quality of the obtained first corpus and second corpus.

[0048] Please refer to Figure 3 , Figure 3A flowchart of a method for determining the semantic matching probability of the target corpus pair using a preset translation model provided in this embodiment, namely, for Figure 2 Step 203 in the process 200 shown provides a specific implementation method. The other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced with step 203 to obtain a new complete embodiment. The process 300 includes the following steps:

[0049] Step 301: Input the target corpus into a preset first translation model to obtain output reference first corpus and reference second corpus.

[0050] In this embodiment, the target corpus pair is input into a first translation model, which is used to translate the corpus between the two languages ​​corresponding to the first corpus and the second corpus, respectively, and the translation confidence of the first translation model meets the pre-configured confidence requirement. The translation model is used to generate a reference second corpus for the first corpus in the language of the second corpus, and to generate a reference first corpus for the second corpus in the language of the first corpus.

[0051] Step 302: Generate a forward translation probability based on the similarity between the second corpus in the target corpus pair and the reference second corpus.

[0052] In this embodiment, the forward translation probability is determined based on the similarity between the reference second corpus and the second corpus, wherein the similarity is generally the similarity in text content between the second corpus and the reference second corpus. Furthermore, it can be configured that when the similarity in text content is lower than a preset similarity threshold, the semantic similarity between the second corpus and the reference second corpus is obtained, and the above two types of similarities are summarized based on average processing, weighted processing, etc., to obtain the forward translation probability based on the similarity determined based on the text content and the semantic similarity.

[0053] Step 303: Generate a back-translation probability based on the similarity between the first corpus in the target corpus pair and the reference first corpus.

[0054] In this embodiment, the method for determining the back-translation probability is the same as the method for determining the forward translation probability described in step 302 above. The only difference is that the back-translation probability is generated using the first corpus and the reference corpus. Therefore, the same parts can refer to the contents described in step 302 and will not be repeated here.

[0055] Step 304: Determine the semantic matching probability of the target corpus pair based on the forward translation probability and the backward translation probability.

[0056] In this embodiment, the semantic matching probability of the target corpus pair is obtained by summarizing the forward translation probability and the backward translation probability determined in step 302 and step 303 respectively.

[0057] This implementation Figure 2 On the basis of the illustrated embodiment, further, the first corpus and the second corpus can be translated respectively based on the first translation model whose confidence meets the requirements, and the reference second corpus and the reference first corpus whose confidence meets the requirements are determined, so as to characterize the semantic matching probability based on the similarity between the reference second corpus and the reference first corpus whose confidence meets the requirements and the second corpus and the first corpus, thereby improving the efficiency of determining the semantic matching probability.

[0058] Please refer to Figure 4 , Figure 4 A flowchart of a method for determining the semantic matching probability of the target corpus pair using a preset translation model provided in this embodiment, namely, for Figure 2 Step 203 in the process 200 shown provides a specific implementation method. The other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 203 to obtain a new complete embodiment. The process 400 includes the following steps:

[0059] Step 401: Input the target corpus into a preset second translation model to obtain an output positive association probability.

[0060] In this embodiment, the target corpus pair is input into a preset second translation model, which is used to determine the association probability that the content obtained by translating the first corpus into the language of the second corpus is the second corpus. The second translation model is used to determine the positive association probability that the content obtained by translating the first corpus in the target corpus pair into the language of the second corpus is the second corpus.

[0061] Step 402: Input the target corpus into a preset third translation model to obtain an output reverse correlation probability.

[0062] In this embodiment, the target corpus pair is input into a preset third translation model, which is used to determine the association probability that the content obtained by translating the second corpus into the language of the first corpus is the first corpus, and the third translation model is used to determine the reverse association probability that the content obtained by translating the second corpus in the target corpus pair into the language of the first corpus is the first corpus.

[0063] Step 403: Determine the semantic matching probability of the target corpus pair based on the forward association probability and the reverse association probability.

[0064] In this embodiment, the forward association probability and the reverse association probability obtained in steps 401 and 402 are summarized to obtain the semantic matching probability of the target corpus pair.

[0065] This implementation Figure 2 On the basis of the embodiment shown, further, two unidirectional translation models can be directly used to determine the semantic matching probability of the target corpus pair, so as to reduce the difficulty of training the translation model and avoid the impact of low translation model training quality on the accuracy of determining the semantic matching probability.

[0066] Based on any of the above embodiments, the corpus alignment method further includes: determining the target corpus pair that has completed the corpus alignment process as an aligned corpus pair; and generating a bilingual dictionary for corpus translation between the language of the first corpus and the language of the second corpus based on each aligned corpus pair.

[0067] Specifically, the target corpus pair that has completed the corpus alignment process is determined as the aligned corpus pair, and after summarizing the aligned corpus pairs, a bilingual dictionary is generated for corpus translation between the language of the first corpus and the language of the second corpus, which includes more aligned first and second corpora that are not statistically significant and not recorded.

[0068] Furthermore, in some optional embodiments, the corpus alignment method further includes: respectively acquiring semantic information of each of the aligned corpus pairs, and merging the aligned corpus pairs whose semantic information overlap is greater than a third threshold.

[0069] Specifically, when generating a bilingual dictionary, after obtaining the semantic information of each aligned corpus pair, the overlap of the semantic information of each aligned corpus with the information of other aligned corpora is calculated based on the semantic information of each aligned corpus pair, and two (or more) aligned corpora with the overlap greater than a third threshold are merged, so that semantically similar aligned corpora pairs can be displayed together at similar positions in the bilingual dictionary, thereby improving the usability of the bilingual dictionary.

[0070] To deepen understanding, this disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 5 Flow 500 is shown.

[0071] After obtaining the first corpus "soccer" in the "Chinese" language and the second corpus "soccer", "football", and "volleyball" in the "English" language, the initial corpus pairs "soccer-soccer", "soccer-football", and "soccer-volleyball" are formed.

[0072] The statistical forward conditional probability and reverse conditional probability of each initial corpus pair are determined respectively, where the forward conditional probability and reverse conditional probability of the initial corpus pair "soccer-soccer" are: "soccer→soccer"-95%, "soccer→soccer"-90%; the forward conditional probability and reverse conditional probability of the initial corpus pair "soccer-football" are: "soccer→football"-92%, "football→soccer"-97%; the forward conditional probability and reverse conditional probability of the initial corpus pair "soccer-volleyball" are: "soccer→volleyball"-35%, "volleyball→soccer"-40%. The initial corpus pair "soccer-soccer" and the initial corpus pair "soccer-football" whose forward conditional probability and reverse conditional probability both meet the first threshold requirement (greater than the first threshold) (Chinese→English-80%, English→Chinese-90%) are determined as the target corpus pair.

[0073] The target corpus pair "soccer-soccer" and the target corpus pair "soccer-football" are input into the preset first model to obtain the reference second corpus "soccer" and the reference first corpus "soccer" corresponding to the target corpus pair "soccer-soccer", as well as the reference second corpus "soccer" and the reference first corpus "football" corresponding to the target corpus pair "soccer-football".

[0074] For the target corpus pair "football-soccer", a forward translation probability of 100% is generated based on the similarity between the first corpus and the reference first corpus, and a reverse translation probability of 100% is generated based on the similarity between the second corpus and the reference second corpus, thereby generating a semantic matching probability (100%, 100%) for the target corpus pair "football-soccer".

[0075] For the target corpus pair "football-football", the forward translation probability is generated based on the similarity between the first corpus and the reference first corpus (based on text content 50%, based on semantics 100%, based on the weighted ratio of text content and semantics 1:2, the forward translation probability is determined to be 83%), and the reverse translation probability is generated based on the similarity between the second corpus and the reference second corpus (based on text content 0%, based on semantics 100%, based on the weighted ratio of text content and semantics 1:2, the reverse translation probability is determined to be 66%), and the semantic matching probability of the target corpus pair "football-football" is generated (66%, 83%).

[0076] Determine that the semantic matching probabilities of the target corpus pairs "足球 - soccer" and "足球 - football" are both greater than the second threshold (Chinese → English - 60%, English → Chinese - 60%). Align the first corpus and the second corpus in the target corpus pairs to obtain the aligned corpus pairs "足球 - soccer" and "足球 - football".

[0077] After obtaining the semantic information "足球" of the aligned corpus pairs "足球 - soccer" and "足球 - football" respectively, determine that the semantic overlap degree (100%) between the aligned corpus pairs "足球 - soccer" and "足球 - football" is greater than the third threshold (70%). After merging the aligned corpus pairs "足球 - soccer" and "足球 - football", record "足球" - "soccer", "football" in the bilingual dictionary generated for corpus mutual translation between the language of the first corpus and the language of the second corpus.

[0078] Further refer to Figure 6 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a corpus alignment device. This device embodiment corresponds to Figure 2 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0079] As Figure 6 shown, the corpus alignment device 600 of this embodiment may include: an initial corpus pair acquisition unit 601, a target corpus pair determination unit 602, a semantic matching probability calculation unit 603, and a corpus alignment processing unit 604. Among them, the initial corpus pair acquisition unit 601 is configured to acquire an initial corpus pair composed of a first corpus and a second corpus with different languages; the target corpus pair determination unit 602 is configured to determine the forward conditional probability and the reverse conditional probability of the initial corpus pair, and determine the initial corpus pair with both the forward conditional probability and the reverse conditional probability greater than the first threshold as the target corpus pair; the semantic matching probability calculation unit 603 is configured to use a preset translation model to determine the semantic matching probability of the target corpus pair; the corpus alignment processing unit 604 is configured to perform corpus alignment processing on the first corpus and the second corpus in the target corpus pair with the semantic matching probability greater than the second threshold.

[0080] In this embodiment, in the corpus alignment device 600: the specific processing of the initial corpus pair acquisition unit 601, the target corpus pair determination unit 602, the semantic matching probability calculation unit 603, and the corpus alignment processing unit 604 and the technical effects brought by them can be respectively referred to Figure 2 the relevant descriptions of steps 201 - 204 in the corresponding embodiments, and will not be elaborated here.

[0081] In some optional implementations of this embodiment, the semantic matching probability calculation unit 603 includes: a reference corpus generation subunit, configured to input the target corpus pair into a preset first translation model to obtain output reference first corpus and reference second corpus; wherein the first translation model is used to perform corpus translation between two languages ​​corresponding to the first corpus and the second corpus respectively; a forward translation probability generation subunit, configured to generate a forward translation probability based on the similarity between the second corpus in the target corpus pair and the reference second corpus; a backward translation probability generation subunit, configured to generate a backward translation probability based on the similarity between the first corpus in the target corpus pair and the reference first corpus; and a first semantic matching probability calculation subunit, configured to determine the semantic matching probability of the target corpus pair based on the forward translation probability and the backward translation probability.

[0082] In some optional implementations of this embodiment, the semantic matching probability calculation unit 603 includes: a forward association probability calculation subunit, configured to input the target corpus pair into a preset second translation model to obtain an output forward association probability; wherein the second translation model is used to determine the association probability that the content obtained by translating the first corpus into the language of the second corpus is the second corpus; a reverse association probability calculation subunit, configured to input the target corpus pair into a preset third translation model to obtain an output reverse association probability; wherein the third translation model is used to determine the association probability that the content obtained by translating the second corpus into the language of the first corpus is the first corpus; and a second semantic matching probability calculation subunit, configured to determine the semantic matching probability of the target corpus pair based on the forward association probability and the reverse association probability.

[0083] In some optional implementations of this embodiment, the corpus alignment device 600 further includes: an initial corpus acquisition unit, configured to acquire an initial first corpus and an initial second corpus; a corpus cleaning unit, configured to perform data cleaning on the corpora in the initial first corpus and the initial second corpus, respectively, to obtain a cleaned first corpus and a cleaned second corpus; a corpus word segmentation unit, configured to perform word segmentation on the cleaned first corpus and the cleaned second corpus, respectively, to obtain a processed first corpus and a processed second corpus; and a corpus extraction unit, configured to extract the first corpus and the second corpus from the processed first corpus and the processed second corpus, respectively.

[0084] In some optional implementations of this embodiment, the corpus alignment apparatus 600 further includes: an aligned corpus pair determining unit configured to determine a target corpus pair that has completed the corpus alignment process as an aligned corpus pair; and a bilingual dictionary generating unit configured to generate a bilingual dictionary for corpus translation between the language of the first corpus and the language of the second corpus based on each aligned corpus pair.

[0085] In some optional implementations of this embodiment, the corpus alignment device 600 further includes: an aligned corpus pair merging unit configured to respectively obtain semantic information of each aligned corpus pair and merge the aligned corpus pairs whose semantic information overlap is greater than a third threshold.

[0086] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The corpus alignment apparatus provided by this embodiment introduces a translation model to perform secondary screening of word pairs of the target corpus determined based on the positive conditional probability and the reverse conditional probability, so as to reduce the probability threshold requirement when screening the target corpus pairs based on the positive conditional probability and the reverse conditional probability. Under the premise of ensuring the accuracy of the semantic correspondence between the first corpus and the second corpus, more corpus pairs that are not significant at the statistical level can be recalled, thereby improving the alignment quality of the corpus.

[0087] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0088] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0090] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0091] The computing unit 701 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the corpus alignment method. For example, in some embodiments, the corpus alignment method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the corpus alignment method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the corpus alignment method in any other appropriate manner (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0096] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0097] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private server (VPS) services. Servers may also be classified as distributed system servers or servers integrated with blockchain.

[0098] According to the technical solution of the embodiment of the present disclosure, a translation model is introduced to perform secondary screening of word pairs of the target corpus determined based on the forward conditional probability and the reverse conditional probability, so as to reduce the probability threshold requirement when screening the target corpus pairs based on the forward conditional probability and the reverse conditional probability. Under the premise of ensuring the accuracy of the semantic correspondence between the first corpus and the second corpus, more corpus pairs that are not significant at the statistical level can be recalled, thereby improving the alignment quality of the corpus.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0100] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A corpus alignment method, comprising: Obtaining an initial corpus pair consisting of a first corpus and a second corpus in different languages; Determining a forward conditional probability and a reverse conditional probability of the initial corpus pair, and determining an initial corpus pair for which both the forward conditional probability and the reverse conditional probability are greater than a first threshold as a target corpus pair; The method uses a preset translation model to determine the semantic matching probability of the target corpus pair, including: inputting the target corpus pair into a preset first translation model to obtain outputted reference first corpus and reference second corpus; wherein the first translation model is used to translate corpus between two languages ​​corresponding to the first corpus and the second corpus respectively; generating a forward translation probability according to the similarity between the second corpus in the target corpus pair and the reference second corpus; generating a reverse translation probability according to the similarity between the first corpus in the target corpus pair and the reference first corpus; and determining the target corpus pair based on the forward translation probability and the reverse translation probability. The semantic matching probability of the target corpus pair is determined by inputting the target corpus pair into a preset second translation model, and obtaining an output forward association probability; wherein the second translation model is used to determine the association probability that the content obtained by translating the first corpus into the language of the second corpus is the second corpus; the target corpus pair is input into a preset third translation model, and obtaining an output reverse association probability; wherein the third translation model is used to determine the association probability that the content obtained by translating the second corpus into the language of the first corpus is the first corpus; based on the forward association probability and the reverse association probability, the semantic matching probability of the target corpus pair is determined; Corpus alignment processing is performed on the first corpus and the second corpus in the target corpus pair where the semantic matching probability is greater than the second threshold.

2. The method according to claim 1, further comprising: Obtaining an initial first corpus and an initial second corpus; Performing data cleaning on the corpus in the initial first corpus and the initial second corpus respectively to obtain a cleaned first corpus and a cleaned second corpus; performing word segmentation processing on the cleaned first corpus and the cleaned second corpus respectively to obtain a processed first corpus and a processed second corpus; The first corpus and the second corpus are extracted from the processed first corpus and the processed second corpus respectively.

3. The method according to any one of claims 1 to 2, further comprising: Determining the target corpus pair that has completed the corpus alignment process as an aligned corpus pair; Based on each of the aligned corpus pairs, a bilingual dictionary is generated for corpus translation between the language of the first corpus and the language of the second corpus.

4. The method according to claim 3, further comprising: Semantic information of each of the aligned corpus pairs is obtained respectively, and aligned corpus pairs whose semantic information overlap is greater than a third threshold are merged.

5. A corpus alignment device, comprising: an initial corpus pair acquisition unit configured to acquire an initial corpus pair consisting of a first corpus and a second corpus in different languages; a target corpus pair determining unit, configured to determine a forward conditional probability and a reverse conditional probability of the initial corpus pair, and determine an initial corpus pair for which both the forward conditional probability and the reverse conditional probability are greater than a first threshold as a target corpus pair; The semantic matching probability calculation unit is configured to determine the semantic matching probability of the target corpus pair using a preset translation model, including: a reference corpus generation subunit, configured to input the target corpus pair into a preset first translation model to obtain an output reference first corpus and a reference second corpus; wherein the first translation model is used to perform corpus translation between two languages ​​corresponding to the first corpus and the second corpus respectively; a forward translation probability generation subunit, configured to generate a forward translation probability according to the similarity between the second corpus in the target corpus pair and the reference second corpus; a reverse translation probability generation subunit, configured to generate a reverse translation probability according to the similarity between the first corpus in the target corpus pair and the reference first corpus; a first semantic matching probability calculation subunit, configured to generate a reverse translation probability based on the forward translation probability and the reverse translation probability. The reverse translation probability is used to determine the semantic matching probability of the target corpus pair; or the forward association probability calculation subunit is configured to input the target corpus pair into a preset second translation model to obtain an output forward association probability; wherein the second translation model is used to determine the association probability that the content obtained by translating the first corpus into the language of the second corpus is the second corpus; the reverse association probability calculation subunit is configured to input the target corpus pair into a preset third translation model to obtain an output reverse association probability; wherein the third translation model is used to determine the association probability that the content obtained by translating the second corpus into the language of the first corpus is the first corpus; the second semantic matching probability calculation subunit is configured to determine the semantic matching probability of the target corpus pair based on the forward association probability and the reverse association probability; The corpus alignment processing unit is configured to perform corpus alignment processing on the first corpus and the second corpus in the target corpus pair whose semantic matching probability is greater than a second threshold.

6. The apparatus according to claim 5, further comprising: an initial corpus acquisition unit, configured to acquire an initial first corpus and an initial second corpus; a corpus cleaning unit configured to perform data cleaning on the corpus in the initial first corpus and the initial second corpus, respectively, to obtain a cleaned first corpus and a cleaned second corpus; a corpus word segmentation unit configured to perform word segmentation processing on the cleaned first corpus and the cleaned second corpus respectively to obtain a processed first corpus and a processed second corpus; The corpus extraction unit is configured to extract the first corpus and the second corpus from the processed first corpus set and the processed second corpus set respectively.

7. The apparatus according to any one of claims 5 to 6, further comprising: an aligned corpus pair determining unit, configured to determine the target corpus pair that has completed the corpus alignment process as the aligned corpus pair; The bilingual dictionary generating unit is configured to generate a bilingual dictionary for corpus translation between the language of the first corpus and the language of the second corpus based on each of the aligned corpus pairs.

8. The apparatus according to claim 7, further comprising: The aligned corpus pair merging unit is configured to respectively obtain semantic information of each of the aligned corpus pairs and merge the aligned corpus pairs whose semantic information overlap is greater than a third threshold.

9. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the corpus alignment method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the corpus alignment method according to any one of claims 1 to 4.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the corpus alignment method according to any one of claims 1 to 4.

Images