Tree model-based corpus alignment method and device

Abstract

The application discloses a corpus alignment method and device based on a tree model, and the method comprises the following steps: performing corpus preprocessing on an initial corpus set to obtain a first corpus list and a second corpus list; obtaining a correlation matrix of the initial corpus set according to the forward conditional probability and the reverse conditional probability of each element in the first corpus list and the second corpus list; updating the correlation matrix; when the value of each element in the correlation matrix no longer changes, obtaining a plurality of pre-alignment tree models corresponding to the correlation matrix, and the pre-alignment tree models are used for realizing corpus alignment. Through the calculation of the conditional probability of the initial corpus set, the corresponding correlation matrix is obtained, and the correlation matrix is iteratively updated according to an updating process, so that a plurality of pre-alignment tree models used for realizing corpus alignment in a large-scale corpus are obtained, and the efficiency of corpus alignment is improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a corpus alignment method and apparatus based on a tree model. Background Technology

[0002] In multilingual machine translation corpus alignment systems within specific industry sectors, differences in context and user habits often lead to discrepancies in the translation results of words and phrases from one language to another. Therefore, it is necessary to establish a large-scale multilingual machine translation corpus through corpus alignment across languages ​​to enhance the universality of machine translation systems in specific applications.

[0003] However, the existing methods for building and maintaining corpora involve corpus alignment that is labor-intensive and inefficient. Therefore, improving the efficiency of corpus alignment is of paramount importance. Summary of the Invention

[0004] The technical problem that this invention aims to solve is that the existing methods for establishing and maintaining corpora involve corpus alignment, which requires a lot of work and is not very efficient.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a corpus alignment method based on a tree model, comprising:

[0006] The initial corpus is preprocessed to obtain a first corpus list and a second corpus list;

[0007] Based on the positive and negative conditional probabilities of each element in the first and second corpus lists, the correlation matrix of the initial corpus set is obtained.

[0008] Update the association matrix; when the values ​​of each element in the association matrix no longer change, obtain multiple pre-aligned tree models corresponding to the association matrix, which are used to achieve corpus alignment.

[0009] As an optional implementation, updating the correlation matrix includes:

[0010] Based on the correlation matrix, the corpus is sorted according to the character position of the source language to obtain the third corpus list and the fourth corpus list respectively;

[0011] Based on the third corpus list and the fourth corpus list, a first tree model corresponding to each element in the third corpus list is obtained through a corpus statistical pairing algorithm;

[0012] The corpus statistical pairing algorithm is used to select corpora from the fourth corpus list that correspond to each element in the third corpus list, and use them as nodes of the first tree model.

[0013] According to the data statistical decomposition algorithm, the first tree model is processed to obtain a second tree model corresponding to each of the first tree models; and, according to the second tree model, the main corpus corresponding to each element in the third corpus list is obtained;

[0014] Based on the character positions of the source language, the correlation matrix is ​​expanded accordingly, and the elements in the correlation matrix are updated.

[0015] As an optional implementation, the step of processing the first tree model according to a data statistical decomposition algorithm to obtain a second tree model corresponding to each of the first tree models includes:

[0016] Based on the paths from each leaf node to each root node, the growth direction of the corpus corresponding to each node in the first tree model is determined, and based on the data statistical decomposition algorithm, the statistical frequency of each node in the first tree model corresponding to different growth directions is determined, thereby obtaining the second tree model corresponding to the first tree model.

[0017] The step of obtaining the main corpus corresponding to each element in the third corpus list based on the second tree model includes:

[0018] Determine at least one second tree model corresponding to each element in the third corpus list;

[0019] Based on the directed path from each leaf to the root node in the second tree model, the node in each directed tree model is labeled with a subject word, and the corpus corresponding to the labeled node is determined as the main corpus.

[0020] As an optional implementation, the step of tagging the node in each of the directed tree models with a subject term based on the directed path from each leaf to the root node in the second tree model includes:

[0021] Determine the connection relationships between multiple second tree models, and divide the multiple second tree models into multiple subtree models according to the connection relationships;

[0022] For the multiple subtree models, based on the directed path from the leaf to the root node in each subtree model, determine the non-bottom node with the highest statistical frequency on the directed path corresponding to the same root node, and label the non-bottom node with a subject term.

[0023] As an optional implementation, after determining the non-bottom-level node with the highest statistical frequency on the directed path corresponding to the same root node based on the directed path from the leaf to the root node in each of the multiple subtree models, and tagging the non-bottom-level node with a subject term, the method further includes:

[0024] Determine the statistical frequency threshold that matches the subject tag;

[0025] If at least two root nodes in the subtree model have the subject tag, then according to the statistical frequency threshold, the subject tags on nodes with a statistical frequency lower than the statistical frequency threshold are removed.

[0026] As an optional implementation, the step of performing comparative dilation on the association matrix based on the character positions of the source language and updating the elements in the association matrix includes:

[0027] Based on the character positions in the source language, the second tree model is used to determine the candidate target corpus corresponding to each corpus in the source language;

[0028] The alignment probability between each corpus in the source language and the candidate target corpus is determined, and the candidate target corpus whose alignment probability exceeds a preset alignment probability threshold is determined as the target corpus corresponding to each corpus in the source language.

[0029] As an optional implementation, the method further includes:

[0030] Determine the language of the corpus to be translated, and determine whether the language of the corpus to be translated is consistent with the source language or the target language;

[0031] If it is determined that the language of the corpus to be translated is consistent with the source language or the target language, then according to the pre-alignment tree model, the corpus to be translated is aligned, and the corpus alignment result with the highest global probability is output.

[0032] Otherwise, after pre-translating the corpus to be translated into the source language or the target language, the corpus to be translated after pre-translation is aligned according to the pre-alignment tree model, and the corpus alignment result with the highest global probability is output;

[0033] The global probability is used to indicate the overall corpus alignment probability obtained by a preset algorithm for each subcorpus of the corpus to be translated arranged in sequence.

[0034] A second aspect of the present invention provides a corpus alignment device based on a tree model, the device comprising:

[0035] The corpus preprocessing module is used to preprocess the initial corpus set to obtain the first corpus list and the second corpus list;

[0036] The association matrix generation module is used to obtain the association matrix of the initial corpus set based on the positive conditional probability and the negative conditional probability of each element in the first corpus list and the second corpus list.

[0037] The association matrix update module is used to update the association matrix; when the values ​​of each element in the association matrix no longer change, multiple pre-aligned tree models corresponding to the association matrix are obtained, and the pre-aligned tree models are used to achieve corpus alignment.

[0038] As an optional implementation, the specific method by which the association matrix update module updates the association matrix includes:

[0039] Based on the correlation matrix, the corpus is sorted according to the character position of the source language to obtain the third corpus list and the fourth corpus list respectively;

[0040] Based on the third corpus list and the fourth corpus list, a first tree model corresponding to each element in the third corpus list is obtained through a corpus statistical pairing algorithm;

[0041] The corpus statistical pairing algorithm is used to select corpora from the fourth corpus list that correspond to each element in the third corpus list, and use them as nodes of the first tree model.

[0042] According to the data statistical decomposition algorithm, the first tree model is processed to obtain a second tree model corresponding to each of the first tree models; and, according to the second tree model, the main corpus corresponding to each element in the third corpus list is obtained;

[0043] Based on the character positions of the source language, the correlation matrix is ​​expanded accordingly, and the elements in the correlation matrix are updated.

[0044] As an optional implementation, the correlation matrix update module processes the first tree model according to a data statistical decomposition algorithm to obtain a second tree model corresponding to each of the first tree models in a specific way, including:

[0045] Based on the paths from each leaf node to each root node, the growth direction of the corpus corresponding to each node in the first tree model is determined, and based on the data statistical decomposition algorithm, the statistical frequency of each node in the first tree model corresponding to different growth directions is determined, thereby obtaining the second tree model corresponding to the first tree model.

[0046] The correlation matrix update module obtains the specific method of the main corpus corresponding to each element in the third corpus list based on the second tree model, including:

[0047] Determine at least one second tree model corresponding to each element in the third corpus list;

[0048] Based on the directed path from each leaf to the root node in the second tree model, the node in each directed tree model is labeled with a subject word, and the corpus corresponding to the labeled node is determined as the main corpus.

[0049] As an optional implementation, the specific method by which the association matrix update module performs subject tagging on each node in the directed tree model based on the directed path from each leaf to the root node in the second tree model includes:

[0050] Determine the connection relationships between multiple second tree models, and divide the multiple second tree models into multiple subtree models according to the connection relationships;

[0051] For the multiple subtree models, based on the directed path from the leaf to the root node in each subtree model, determine the non-bottom node with the highest statistical frequency on the directed path corresponding to the same root node, and label the non-bottom node with a subject term.

[0052] As an optional implementation, the apparatus further includes a filtering module, used by the association matrix update module to determine, for each of the plurality of subtree models, the non-bottom-level node with the highest statistical frequency on the directed path corresponding to the same root node, based on the directed path from the leaf to the root node in each subtree model, and then to label the non-bottom-level node with a subject term.

[0053] Determine the statistical frequency threshold that matches the subject tag;

[0054] If at least two root nodes in the subtree model have the subject tag, then according to the statistical frequency threshold, the subject tags on nodes with a statistical frequency lower than the statistical frequency threshold are removed.

[0055] As an optional implementation, the correlation matrix update module performs comparative expansion on the correlation matrix based on the character positions of the source language, and updates the elements in the correlation matrix in the following specific ways:

[0056] Based on the character positions in the source language, the second tree model is used to determine the candidate target corpus corresponding to each corpus in the source language;

[0057] The alignment probability between each corpus in the source language and the candidate target corpus is determined, and the candidate target corpus whose alignment probability exceeds a preset alignment probability threshold is determined as the target corpus corresponding to each corpus in the source language.

[0058] As an optional implementation, the apparatus further includes a corpus alignment module for:

[0059] Determine the language of the corpus to be translated, and determine whether the language of the corpus to be translated is consistent with the source language or the target language;

[0060] If it is determined that the language of the corpus to be translated is consistent with the source language or the target language, then according to the pre-alignment tree model, the corpus to be translated is aligned, and the corpus alignment result with the highest global probability is output.

[0061] Otherwise, after pre-translating the corpus to be translated into the source language or the target language, the corpus to be translated after pre-translation is aligned according to the pre-alignment tree model, and the corpus alignment result with the highest global probability is output;

[0062] The global probability is used to indicate the overall corpus alignment probability obtained by a preset algorithm for each subcorpus of the corpus to be translated arranged in sequence.

[0063] A third aspect of the present invention discloses another tree-based corpus alignment device, the device comprising:

[0064] Memory containing executable program code;

[0065] A processor coupled to the memory;

[0066] The processor calls the executable program code stored in the memory to execute the tree-based corpus alignment method disclosed in the first aspect of the present invention.

[0067] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the tree-based corpus alignment method disclosed in the first aspect of the present invention.

[0068] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by calculating the conditional probability of the initial corpus, the corresponding correlation matrix is ​​obtained, and the correlation matrix is ​​iteratively updated according to the update process, thereby obtaining multiple pre-aligned tree models for corpus alignment in a large-scale corpus, thereby improving the efficiency of corpus alignment. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating a tree-based corpus alignment method disclosed in Embodiment 1 of the present invention.

[0071] Figure 2 This is a schematic diagram illustrating an application scenario of a tree-based corpus alignment method disclosed in Embodiment 1 of the present invention;

[0072] Figure 3 This is a flowchart illustrating a tree-based corpus alignment method disclosed in Embodiment 2 of the present invention.

[0073] Figure 4 This is a schematic diagram of a tree-model-based corpus alignment device disclosed in Embodiment 3 of the present invention;

[0074] Figure 5 This is a schematic diagram of another tree-model-based corpus alignment device disclosed in Embodiment 3 of the present invention;

[0075] Figure 6 This is a schematic diagram of a tree-based corpus alignment device disclosed in Embodiment 4 of the present invention. Detailed Implementation

[0076] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0078] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0079] In multilingual machine translation corpus alignment systems within specific industry sectors, differences in context and user habits often lead to discrepancies in the translation results of words and phrases from one language to another. Therefore, it is necessary to establish a large-scale multilingual machine translation corpus through corpus alignment across languages ​​to enhance the universality of machine translation systems in specific applications.

[0080] However, the existing methods for building and maintaining corpora involve corpus alignment that is labor-intensive and inefficient. Therefore, improving the efficiency of corpus alignment is of paramount importance.

[0081] The tree-based corpus alignment method proposed in this application can be used for bilingual corpus pre-alignment processing in machine translation systems. The relational network generated based on the optimal solution of conditional statistical probability, or tree model, can also effectively solve problems such as polysemous words and extended words in the field of machine translation. In the maintenance of standardized word libraries, batch standardization verification can be performed, improving the efficiency of manual corpus alignment and reducing manual costs.

[0082] This invention discloses a corpus alignment method and apparatus based on a tree model. By calculating the conditional probability of an initial corpus set, a corresponding correlation matrix is ​​obtained. The correlation matrix is ​​then iteratively updated according to an update process, resulting in multiple pre-aligned tree models for corpus alignment in a large-scale corpus, thereby improving the efficiency of corpus alignment.

[0083] Example 1

[0084] Please see Figure 1 , Figure 1 This is a flowchart illustrating a tree-model-based corpus alignment method disclosed in Embodiment 1 of the present invention. Figure 1 As shown, the tree-based corpus alignment method may include the following operations:

[0085] S101. Perform corpus preprocessing on the initial corpus set to obtain the first corpus list and the second corpus list;

[0086] The initial corpus may include two or more languages. When the initial corpus includes multiple languages, the method provided in this application can be used to generate multiple sets of pre-aligned tree models corresponding to these languages, forming a pre-aligned tree network. Those skilled in the art can deduce this without any inventive effort, and this should not be considered a limitation. To ensure the readability and understandability of this application, each embodiment is described using an initial corpus containing only two languages ​​as an example.

[0087] S102. Based on the positive conditional probability and negative conditional probability of each element in the first corpus list and the second corpus list, obtain the correlation matrix of the initial corpus set.

[0088] After preprocessing the initial corpus, it can be divided into a first corpus list corresponding to the source language and a second corpus list corresponding to the target language. Once the corresponding corpus lists are obtained, the forward and backward conditional probabilities of each corpus can be calculated to generate an association matrix for the initial corpus. The elements in the association matrix represent the forward and backward conditional probabilities arranged according to the order of the corpus data. Correspondingly, when the initial corpus includes three or more languages, a corresponding three-dimensional or more-dimensional association matrix can also be generated.

[0089] S103. Update the association matrix; when the values ​​of each element in the association matrix no longer change, obtain multiple pre-aligned tree models corresponding to the association matrix, and the pre-aligned tree models are used to achieve corpus alignment.

[0090] According to the method described in the next embodiment, the correlation matrix can be updated in real time. When the values ​​of each element in the correlation matrix no longer change, it indicates that convergence has been achieved. At this time, training is completed, and multiple pre-aligned tree models corresponding to the correlation matrix are obtained. The pre-aligned tree models are used to achieve corpus alignment.

[0091] As an optional implementation, updating the correlation matrix includes:

[0092] Based on the association matrix, the corpus is sorted according to the character positions of the source language to obtain a third corpus list and a fourth corpus list respectively; the third corpus list and the fourth corpus list are determined according to the association matrix to generate a third corpus list corresponding to the source language and a fourth corpus list corresponding to the target language.

[0093] Based on the third corpus list and the fourth corpus list, a first tree model corresponding to each element in the third corpus list is obtained through a corpus statistical pairing algorithm;

[0094] The corpus statistical pairing algorithm is used to select corpora from the fourth corpus list that correspond to each element in the third corpus list, and use them as nodes of the first tree model. Simultaneously, each node of the first tree model has a corresponding association statistical frequency. For example, based on the association matrix and the initial corpus set, in the first tree model corresponding to "bank", the association statistical frequency of "cylinder" is 240, indicating that in the initial corpus set, "bank" in the source language appears 240 times when translated into the target language as "cylinder".

[0095] According to the data statistical decomposition algorithm, the first tree model is processed to obtain a second tree model corresponding to each of the first tree models; and, according to the second tree model, the main corpus corresponding to each element in the third corpus list is obtained;

[0096] Based on the character positions of the source language, the correlation matrix is ​​expanded accordingly, and the elements in the correlation matrix are updated.

[0097] The second tree model starts from the leaf node of the corresponding first tree model, expands upwards in both left and right directions, and moves backwards along the path towards the root node, performing data statistical decomposition to generate a directed tree model. According to methods provided in other implementations, the main corpus corresponding to each element in the third corpus list can be determined thereby. Then, based on the character positions of the source language, the association matrix is ​​expanded by comparison, and the elements in the association matrix are updated. This process is iterated repeatedly until the elements of the association matrix no longer change.

[0098] By using the association matrix, sorted lists of corpora corresponding to the source and target languages ​​are obtained respectively. One or more first tree models corresponding to each element in the third corpus list are obtained through statistical pairing methods. A corresponding directed tree is obtained based on each first tree model. The main corpus is determined and the elements of the association matrix are updated based on the directed trees. Multiple pre-aligned tree models for corpus alignment in a large-scale corpus are obtained, thereby improving the efficiency of corpus alignment.

[0099] As an optional implementation, the step of processing the first tree model according to a data statistical decomposition algorithm to obtain a second tree model corresponding to each of the first tree models includes:

[0100] Based on the paths from each leaf node to each root node, the growth direction of the corpus corresponding to each node in the first tree model is determined, and based on the data statistical decomposition algorithm, the statistical frequency of each node in the first tree model corresponding to different growth directions is determined, thereby obtaining the second tree model corresponding to the first tree model.

[0101] There are typically only two growth directions: adding corpora before or after the corresponding corpus, which can be referred to as "left growth" and "right growth," respectively. For example, in the case of Chinese as the target language, "device-driven" corpora are obtained through right growth to become "device-driver," while "driver" corpora are obtained through left growth to become "device-driver." Accordingly, both "left growth" and "right growth" have their own statistical frequencies. When determining the reverse path from the leaf node to the root node, the statistical frequency depends on the corresponding growth direction.

[0102] The step of obtaining the main corpus corresponding to each element in the third corpus list based on the second tree model includes:

[0103] Determine at least one second tree model corresponding to each element in the third corpus list;

[0104] Based on the directed path from each leaf to the root node in the second tree model, the node in each directed tree model is labeled with a subject word, and the corpus corresponding to the labeled node is determined as the main corpus.

[0105] The subject tagging identifies the main corpus of the target language within the one or more second-tree models. For specific determination methods, please refer to other implementation methods.

[0106] By determining the growth direction of each node in the first tree model and the statistical frequency corresponding to different growth directions, a directed second tree model corresponding to the first tree model is obtained. The main corpus is then determined based on the determined second tree model, which improves the efficiency and accuracy of corpus alignment.

[0107] As an optional implementation, the step of tagging the node in each of the directed tree models with a subject term based on the directed path from each leaf to the root node in the second tree model includes:

[0108] Determine the connection relationships between multiple second tree models, and divide the multiple second tree models into multiple subtree models according to the connection relationships;

[0109] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of a tree-model-based corpus alignment method disclosed in Embodiment 1 of the present invention. For example... Figure 2 As shown, the diagram includes multiple first-tree models corresponding to the source language corpus "driver" and corresponding second-tree models below them. Based on the connection relationships between the multiple second-tree models, they can be divided into multiple subtree models. Figure 2 The second tree model in the text can divide two unconnected parts into two subtree models. The left subtree model can also be divided into two subtree models according to the growth direction of the "side" in the corpus.

[0110] For the multiple subtree models, based on the directed path from the leaf to the root node in each subtree model, determine the non-bottom node with the highest statistical frequency on the directed path corresponding to the same root node, and label the non-bottom node with a subject term.

[0111] exist Figure 2 In the scenario shown, "driver", "driver's seat", "driver" and "driver" can all be tagged as subjects.

[0112] By determining the relationship between multiple second tree models corresponding to the same element in the third corpus list, more accurate subject tagging is achieved, improving the efficiency and accuracy of corpus alignment.

[0113] As an optional implementation, after determining the non-bottom-level node with the highest statistical frequency on the directed path corresponding to the same root node based on the directed path from the leaf to the root node in each of the multiple subtree models, and tagging the non-bottom-level node with a subject term, the method further includes:

[0114] Determine the statistical frequency threshold that matches the subject tag;

[0115] If at least two root nodes in the subtree model have the subject tag, then according to the statistical frequency threshold, the subject tags on nodes with a statistical frequency lower than the statistical frequency threshold are removed.

[0116] exist Figure 2 In the scenario shown, by determining the threshold, each subtree model can be limited to only one node for subject tagging. Therefore, the subject tag for "driver's seat" can be removed.

[0117] By filtering by statistical frequency after determining the subject markers and removing subject markers that do not meet the criteria, more accurate subject markers are achieved, thus improving the efficiency and accuracy of corpus alignment.

[0118] As an optional implementation, the step of performing comparative dilation on the association matrix based on the character positions of the source language and updating the elements in the association matrix includes:

[0119] Based on the character positions in the source language, the second tree model is used to determine the candidate target corpus corresponding to each corpus in the source language;

[0120] After determining the second tree model, it is also necessary to determine the candidate target corpus corresponding to each corpus in the source language in the second tree model based on the character positions of the source language. The candidate target corpus can also be understood as the initial matching result of the source language corpus.

[0121] The alignment probability between each corpus in the source language and the candidate target corpus is determined, and the candidate target corpus whose alignment probability exceeds a preset alignment probability threshold is determined as the target corpus corresponding to each corpus in the source language.

[0122] If each corpus in the source language consists of a single word, the alignment probability can be one-to-one. However, if each corpus in the source language consists of short or long sentences, then the "global probability" described in Example 2 needs to be considered to address issues such as missing words or untranslated text. There can be only one target corpus, or multiple target corpora can be provided for selection based on the actual application scenario.

[0123] By determining the character positions in the source language and the corresponding multiple second-tree models, candidate target corpora corresponding to each corpus in the source language are identified. Among the candidate target corpora, target corpora that meet the alignment probability requirements are selected. This approach can comprehensively address the issue of simultaneous translation of multiple corpora, thereby improving the efficiency and accuracy of corpus alignment.

[0124] This embodiment provides a tree-based corpus alignment method. The method includes: preprocessing an initial corpus to obtain a first corpus list and a second corpus list; obtaining an association matrix of the initial corpus based on the forward and backward conditional probabilities of each element in the first and second corpus lists; updating the association matrix; and obtaining multiple pre-alignment tree models corresponding to the association matrix when the values ​​of each element in the association matrix no longer change. These pre-alignment tree models are used to achieve corpus alignment. By calculating the conditional probabilities of the initial corpus to obtain the corresponding association matrix and iteratively updating the association matrix according to the update process, multiple pre-alignment tree models for achieving corpus alignment in a large-scale corpus are obtained, thereby improving the efficiency of corpus alignment.

[0125] Example 2

[0126] Please see Figure 3 , Figure 3 This is a flowchart illustrating a tree-model-based corpus alignment method disclosed in Embodiment 2 of the present invention. Figure 3 As shown, based on any other embodiment, the method further includes:

[0127] S201. Determine the language of the corpus to be translated, and determine whether the language of the corpus to be translated is consistent with the source language or the target language;

[0128] This method generates multiple pre-aligned tree models corresponding to different source-target language pairing groups based on the matching method between the source and target languages, and solves the corpus alignment problem through these pre-aligned tree models. However, in applications, some technical fields typically only have a few major languages. When the language of the corpus to be translated is inconsistent with the languages ​​that the determined pre-aligned tree models can align, there may be obstacles to its application. For example, in the automotive field, several pre-aligned tree models for pairwise corpus alignment of Chinese, German, English, Japanese, etc., can be established according to the method in Embodiment 1, or a multi-dimensional model can be established to realize a pre-alignment network for mutual translation between three or more languages. However, for languages ​​outside the corpus, such as Russian, there are difficulties in applying the tree model-based corpus alignment method, requiring preprocessing or conversion to other methods for application.

[0129] S202. If it is determined that the language of the corpus to be translated is consistent with the source language or the target language, then according to the pre-alignment tree model, the corpus to be translated is aligned, and the corpus alignment result with the highest global probability is output.

[0130] For example, according to the method provided in this application, multiple pre-aligned tree models for Chinese-English translation are pre-established. When the corpus to be translated is Chinese and the target language corresponding to the corpus to be translated is English, or when the corpus to be translated is English and the target language corresponding to the corpus to be translated is Chinese, the pre-aligned tree models generated from the Chinese-English corpus by the method of this application can be directly applied to align the corpus to be translated and output the corpus alignment result with the highest global probability.

[0131] S203. Otherwise, after pre-translating the corpus to be translated into the source language or the target language, the corpus to be translated after pre-translation is aligned according to the pre-alignment tree model, and the corpus alignment result with the highest global probability is output.

[0132] As in the application scenario S202, if multiple pre-aligned tree models for Chinese-English translation are pre-established, but the current corpus to be translated is German, then it is necessary to translate the German text to the source language using other methods, and then translate the pre-translated text to the target language using the multiple pre-aligned tree models, or directly translate it to the target language. If the pre-translated text is directly translated to the target language, it can also be converted to the source language through corpus alignment, and then the text to be translated is translated to the source language. Further corpus alignment and global probability calculation are performed using the source and target languages ​​corresponding to the corpus to be translated to verify the translation effect and determine whether other pre-aligned tree models need to be used.

[0133] In this scenario, a chain-like corpus alignment method can also be used to output the corpus alignment results. For example, if the current goal is to translate German into Chinese, but only multiple pre-aligned tree models for German-English and Chinese-English translation are available, then the multiple pre-aligned tree models for German-English translation can be used as a springboard. First, the German corpus to be translated is aligned, and the English corpus alignment result with the highest global probability is output. Then, the translated text is translated into Chinese through the multiple pre-aligned tree models for Chinese-English translation, and the Chinese corpus alignment result with the highest global probability is output.

[0134] The global probability refers to the overall alignment probability of the corpus of words to be translated, obtained through a preset algorithm, for each sub-corpus arranged in sequence. The global probability considers sentences or paragraphs as a whole, thus addressing issues such as missing words or omissions.

[0135] The translation of corpora in various languages ​​was carried out by using the determined pre-aligned tree model, which improved the efficiency and practicality of corpus alignment.

[0136] Example 3

[0137] Embodiment 3 of the present invention also provides a corpus alignment device based on a tree model to implement the aforementioned method. Please refer to [link to related documentation]. Figure 4 , Figure 4 This is a schematic diagram of a tree-model-based corpus alignment device disclosed in Embodiment 3 of the present invention. Figure 4 As shown, based on any other embodiment, the apparatus includes:

[0138] Corpus preprocessing module 31 is used to preprocess the initial corpus set to obtain a first corpus list and a second corpus list;

[0139] The association matrix generation module 32 is used to obtain the association matrix of the initial corpus set based on the positive conditional probability and the negative conditional probability of each element in the first corpus list and the second corpus list.

[0140] The association matrix update module 33 is used to update the association matrix; when the values ​​of each element in the association matrix no longer change, multiple pre-aligned tree models corresponding to the association matrix are obtained, and the pre-aligned tree models are used to achieve corpus alignment.

[0141] By calculating the conditional probability of the initial corpus, the corresponding correlation matrix is ​​obtained. The correlation matrix is ​​then iteratively updated according to the update process, resulting in multiple pre-aligned tree models for corpus alignment in a large-scale corpus, thereby improving the efficiency of corpus alignment.

[0142] As an optional implementation, the specific method by which the association matrix update module 33 updates the association matrix includes:

[0143] Based on the correlation matrix, the corpus is sorted according to the character position of the source language to obtain the third corpus list and the fourth corpus list respectively;

[0144] Based on the third corpus list and the fourth corpus list, a first tree model corresponding to each element in the third corpus list is obtained through a corpus statistical pairing algorithm;

[0145] The corpus statistical pairing algorithm is used to select corpora from the fourth corpus list that correspond to each element in the third corpus list, and use them as nodes of the first tree model.

[0146] According to the data statistical decomposition algorithm, the first tree model is processed to obtain a second tree model corresponding to each of the first tree models; and, according to the second tree model, the main corpus corresponding to each element in the third corpus list is obtained;

[0147] Based on the character positions of the source language, the correlation matrix is ​​expanded accordingly, and the elements in the correlation matrix are updated.

[0148] By using the association matrix, sorted lists of corpora corresponding to the source and target languages ​​are obtained respectively. One or more first tree models corresponding to each element in the third corpus list are obtained through statistical pairing methods. A corresponding directed tree is obtained based on each first tree model. The main corpus is determined and the elements of the association matrix are updated based on the directed trees. Multiple pre-aligned tree models for corpus alignment in a large-scale corpus are obtained, thereby improving the efficiency of corpus alignment.

[0149] As an optional implementation, the correlation matrix update module 33 processes the first tree model according to a data statistical decomposition algorithm to obtain a specific method for obtaining a second tree model corresponding to each of the first tree models, including:

[0150] Based on the paths from each leaf node to each root node, the growth direction of the corpus corresponding to each node in the first tree model is determined, and based on the data statistical decomposition algorithm, the statistical frequency of each node in the first tree model corresponding to different growth directions is determined, thereby obtaining the second tree model corresponding to the first tree model.

[0151] The association matrix update module 33 obtains the specific method of the main corpus corresponding to each element in the third corpus list based on the second tree model, including:

[0152] Determine at least one second tree model corresponding to each element in the third corpus list;

[0153] Based on the directed path from each leaf to the root node in the second tree model, the node in each directed tree model is labeled with a subject word, and the corpus corresponding to the labeled node is determined as the main corpus.

[0154] By determining the growth direction of each node in the first tree model and the statistical frequency corresponding to different growth directions, a directed second tree model corresponding to the first tree model is obtained. The main corpus is then determined based on the determined second tree model, which improves the efficiency and accuracy of corpus alignment.

[0155] As an optional implementation, the association matrix update module 33, based on the directed path from each leaf to the root node in the second tree model, performs subject tagging on each node in the directed tree model in a specific way, including:

[0156] Determine the connection relationships between multiple second tree models, and divide the multiple second tree models into multiple subtree models according to the connection relationships;

[0157] For the multiple subtree models, based on the directed path from the leaf to the root node in each subtree model, determine the non-bottom node with the highest statistical frequency on the directed path corresponding to the same root node, and label the non-bottom node with a subject term.

[0158] By determining the relationship between multiple second tree models corresponding to the same element in the third corpus list, more accurate subject tagging is achieved, improving the efficiency and accuracy of corpus alignment.

[0159] Please see Figure 5 , Figure 5 This is a schematic diagram of another tree-model-based corpus alignment device disclosed in Embodiment 3 of the present invention. Figure 5 As shown, as an optional implementation, the device further includes a filtering module 34, used by the association matrix update module 33 to determine, in each of the multiple subtree models, the non-bottom-level node with the highest statistical frequency on the directed path from the leaf to the root node corresponding to the same root node, based on the directed path from the leaf to the root node in each subtree model, and then marking the non-bottom-level node with a subject term.

[0160] Determine the statistical frequency threshold that matches the subject tag;

[0161] If at least two root nodes in the subtree model have the subject tag, then according to the statistical frequency threshold, the subject tags on nodes with a statistical frequency lower than the statistical frequency threshold are removed.

[0162] By filtering by statistical frequency after determining the subject markers and removing subject markers that do not meet the criteria, more accurate subject markers are achieved, thus improving the efficiency and accuracy of corpus alignment.

[0163] As an optional implementation, the correlation matrix update module 33 updates the elements in the correlation matrix by performing comparative expansion based on the character positions of the source language, including:

[0164] Based on the character positions in the source language, the second tree model is used to determine the candidate target corpus corresponding to each corpus in the source language;

[0165] The alignment probability between each corpus in the source language and the candidate target corpus is determined, and the candidate target corpus whose alignment probability exceeds a preset alignment probability threshold is determined as the target corpus corresponding to each corpus in the source language.

[0166] By determining the character positions in the source language and the corresponding multiple second-tree models, candidate target corpora corresponding to each corpus in the source language are identified. Among the candidate target corpora, target corpora that meet the alignment probability requirements are selected. This approach can comprehensively address the issue of simultaneous translation of multiple corpora, thereby improving the efficiency and accuracy of corpus alignment.

[0167] like Figure 5 As shown, in an optional implementation, the device further includes a corpus alignment module 35, used for:

[0168] Determine the language of the corpus to be translated, and determine whether the language of the corpus to be translated is consistent with the source language or the target language;

[0169] If it is determined that the language of the corpus to be translated is consistent with the source language or the target language, then according to the pre-alignment tree model, the corpus to be translated is aligned, and the corpus alignment result with the highest global probability is output.

[0170] Otherwise, after pre-translating the corpus to be translated into the source language or the target language, the corpus to be translated after pre-translation is aligned according to the pre-alignment tree model, and the corpus alignment result with the highest global probability is output;

[0171] The global probability is used to indicate the overall corpus alignment probability obtained by a preset algorithm for each subcorpus of the corpus to be translated arranged in sequence.

[0172] The translation of corpora in various languages ​​was carried out by using the determined pre-aligned tree model, which improved the efficiency and practicality of corpus alignment.

[0173] Example 4

[0174] Please see Figure 6 , Figure 6 This is a schematic diagram of a tree-model-based corpus alignment device disclosed in Embodiment 4 of the present invention. Figure 6 As shown, the tree-based corpus alignment device may include:

[0175] The device includes a processor 291 and a memory 292 storing executable program code; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 is coupled to the memory 292, and the processor 291 can call logical instructions (executable program code) in the memory 292 to execute the tree-based corpus alignment method described in any of the above embodiments.

[0176] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0177] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0178] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0179] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when invoked, are used to implement the method described in any of the embodiments.

[0180] This invention also discloses a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the steps in the tree-based corpus alignment method described in any embodiment.

[0181] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0182] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0183] Finally, it should be noted that the tree-model-based corpus alignment method and apparatus disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A corpus alignment method based on a tree model, characterized in that, The method includes: The initial corpus is preprocessed to obtain a first corpus list and a second corpus list; Based on the positive and negative conditional probabilities of each element in the first and second corpus lists, the correlation matrix of the initial corpus set is obtained. Update the association matrix; when the values ​​of each element in the association matrix no longer change, obtain multiple pre-aligned tree models corresponding to the association matrix, the pre-aligned tree models being used to achieve corpus alignment; Updating the correlation matrix includes: Based on the correlation matrix, the corpus is sorted according to the character position of the source language to obtain the third corpus list and the fourth corpus list respectively; Based on the third corpus list and the fourth corpus list, a first tree model corresponding to each element in the third corpus list is obtained through a corpus statistical pairing algorithm; the corpus statistical pairing algorithm is used to select corpus data that has a corresponding relationship with each element in the third corpus list from the fourth corpus list as nodes of the first tree model; According to the data statistical decomposition algorithm, the first tree model is processed to obtain a second tree model corresponding to each of the first tree models; and, according to the second tree model, the main corpus corresponding to each element in the third corpus list is obtained; Based on the character positions in the source language, the second tree model is used to determine the candidate target corpus corresponding to each corpus in the source language; The alignment probability between each corpus in the source language and the candidate target corpus is determined, and the candidate target corpus whose alignment probability exceeds a preset alignment probability threshold is determined as the target corpus corresponding to each corpus in the source language.

2. The method according to claim 1, characterized in that, The step of processing the first tree model according to the data statistical decomposition algorithm to obtain a second tree model corresponding to each of the first tree models includes: Based on the paths from each leaf node to each root node, the growth direction of the corpus corresponding to each node in the first tree model is determined, and based on the data statistical decomposition algorithm, the statistical frequency of each node in the first tree model corresponding to different growth directions is determined, thereby obtaining the second tree model corresponding to the first tree model. The step of obtaining the main corpus corresponding to each element in the third corpus list based on the second tree model includes: Determine at least one second tree model corresponding to each element in the third corpus list; Based on the directed path from each leaf to the root node in the second tree model, the nodes in each directed tree model are labeled with subject words, and the corpus corresponding to the labeled nodes is determined as the main corpus.

3. The method according to claim 2, characterized in that, The step of tagging the nodes in each directed tree model with subject terms based on the directed path from each leaf to the root node in the second tree model includes: Determine the connection relationships between multiple second tree models, and divide the multiple second tree models into multiple subtree models according to the connection relationships; For the multiple subtree models, based on the directed path from the leaf to the root node in each subtree model, determine the non-bottom node with the highest statistical frequency on the directed path corresponding to the same root node, and label the non-bottom node with a subject term.

4. The method according to claim 3, characterized in that, The method further includes, for each of the multiple subtree models, determining the non-bottom-level node with the highest statistical frequency on the directed path from the leaf to the root node corresponding to the same root node based on the directed path in each subtree model, and then tagging the non-bottom-level node with the main term. Determine the statistical frequency threshold that matches the subject tag; If at least two root nodes in the subtree model have the subject tag, then according to the statistical frequency threshold, the subject tags on nodes with a statistical frequency lower than the statistical frequency threshold are removed.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Determine the language of the corpus to be translated, and determine whether the language of the corpus to be translated is consistent with the source language or the target language; If it is determined that the language of the corpus to be translated is consistent with the source language or the target language, then according to the pre-alignment tree model, the corpus to be translated is aligned, and the corpus alignment result with the highest global probability is output. Otherwise, after pre-translating the corpus to be translated into the source language or the target language, the corpus to be translated after pre-translation is aligned according to the pre-alignment tree model, and the corpus alignment result with the highest global probability is output; The global probability is used to indicate the overall corpus alignment probability obtained by a preset algorithm for each subcorpus of the corpus to be translated arranged in sequence.

6. A corpus alignment device based on a tree model, characterized in that, The apparatus is used to perform the tree-model-based corpus alignment method as described in any one of claims 1-5, and the apparatus comprises: The corpus preprocessing module is used to preprocess the initial corpus set to obtain the first corpus list and the second corpus list; The association matrix generation module is used to obtain the association matrix of the initial corpus set based on the positive conditional probability and the negative conditional probability of each element in the first corpus list and the second corpus list. The association matrix update module is used to update the association matrix; when the values ​​of each element in the association matrix no longer change, multiple pre-aligned tree models corresponding to the association matrix are obtained, and the pre-aligned tree models are used to achieve corpus alignment.

7. A corpus alignment device based on a tree model, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the tree-based corpus alignment method as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the tree-based corpus alignment method as described in any one of claims 1-5.

Images