Domain bilingual sentence pair selection method and system based on theme information
By using a phrase-pair-based topic model and Jensen-Shannon divergence evaluation, the problem of imbalanced bilingual resource distribution is addressed, resulting in more efficient performance improvement in domain-specific machine translation. Bilingual sentence pairs are selected based on topic relevance, thus expanding the training dataset.
Patent Information
- Application Number
- CN202510947110.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-11-07
AI Technical Summary
The distribution of existing bilingual resources across different fields is extremely uneven, which limits the performance of machine translation in specific fields. Furthermore, existing methods fail to effectively utilize the diversity of topics, resulting in data sparsity and insufficient translation quality.
By constructing a topic distribution of two-sentence pairs based on a phrase pair-based topic model, and using topic vector weighted fusion and Jensen-Shannon divergence evaluation, the sentence pairs most similar to the target domain are selected to expand the training dataset of the machine translation system.
It enables automatic selection of domain-specific two-statement pairs based on topic relevance, improving the performance of machine translation in specific domains and providing more granular methods for domain definition and data augmentation.
Smart Images
Figure CN120911445A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of statistical machine translation, and particularly to a domain bilingual sentence pair selection method based on topic information and a system thereof. BACKGROUND
[0002] The bilingual sentence pair selection method aims to select a subset of sentence pairs related to the domain of the test text from a large-scale bilingual parallel corpus mixed with different domains, to expand the training data set of the specific domain machine translation system, and to improve the performance of the specific domain machine translation. At present, the performance of the machine translation system depends on the quality and size of the training data. The larger the size of the training data and the better the quality, the higher the translation performance. However, the distribution of existing bilingual resources in different domains is extremely unbalanced, and the bilingual resources of some domains are extremely scarce. In addition, the bilingual parallel sentence pairs obtained by the existing bilingual resource mining method usually do not contain corresponding domain labels or do not have a unified domain division method. Therefore, how to automatically mine bilingual sentence pairs related to a specific domain from such a bilingual resource mixed with different domains has become an effective way to improve the performance of the specific domain machine translation.
[0003] The existing bilingual sentence pair selection methods can be roughly divided into three categories: one category is bilingual sentence pair selection method based on information retrieval, which uses test set sentences as queries to retrieve bilingual parallel sentence pairs from general corpus. This method uses test sentences as retrieval input queries, so it cannot meet the needs of fast and real-time translation. The second category is bilingual sentence pair selection method based on language model, which uses target domain monolingual corpus to train domain language model, and evaluates the domain relevance of candidate bilingual sentence pairs according to the cross-entropy of language model. On this basis, researchers propose to use the difference between the cross-entropy of target domain and general domain language models to evaluate the quality of candidate bilingual sentence pairs. In order to alleviate the data sparsity problem of N-gram language model, researchers further use neural network language model cross-entropy difference to evaluate the quality of bilingual sentence pairs. The third category is bilingual sentence pair selection method based on the integration of translation model and language model, which considers both the domain relevance of sentence pairs and the mutual translation of sentence pairs. However, the sentence pair selection methods based on language model or translation model use statistical co-occurrence information of words or n-grams to evaluate the relevance of sentence pairs and target domain. However, due to the size of target domain monolingual or bilingual resources, this method is prone to data sparsity problems. At the same time, the existing methods define "domain" as a collection of homogeneous (same type or source) texts, use all domain data to train translation model or language model, and evaluate the quality of candidate bilingual sentence pairs, ignoring the topic diversity of domain texts. In addition, related work also starts from the granularity of phrase pairs, automatically selects specific types of phrase pairs from general domain phrase table, which can be combined from target domain phrase table, and selects phrase pairs with high domain relevance from these phrase pairs to add to the target domain phrase table.
[0004] Therefore, there is an urgent need to provide a new domain bilingual sentence pair selection method and system based on topic information to solve the above problems. SUMMARY
[0005] The technical problem to be solved by the present application is to provide a domain bilingual sentence pair selection method and system based on topic information, which can evaluate the domain relevance of candidate bilingual sentence pairs according to their relevance to target domain topic distribution, thereby realizing automatic selection of domain bilingual sentence pairs.
[0006] To solve the above technical problems, one technical solution adopted by the present application is to provide a domain bilingual sentence pair selection method based on topic information, comprising the following steps:
[0007] Sl: based on the existing bilingual parallel corpus, all bilingual sentence pairs are processed by word alignment, and phrase pairs satisfying word alignment consistency are extracted from the word-aligned bilingual sentence pairs by using phrase extraction algorithm;
[0008] S2: randomly sampling phrase pairs, and finding all bilingual parallel sentence pairs in which the phrase pairs appear in the bilingual corpus, merging all words in the sentence pairs except the phrase pairs into a document, referred to as a distribution pseudo-document of the phrase pair;
[0009] S3: constructing a phrase pair based topic model, taking the distribution pseudo-document of the phrase pair as input of the topic model after removing stop words, low-frequency words and special characters, training the topic model to obtain the topic distribution of the document, and taking the topic distribution of the document as the topic distribution of the corresponding phrase pair;
[0010] S4: developing a set of phrase pairs extracted from the candidate bilingual sentence pairs and the target field, inferring the topic distribution of the current phrase pair by using the phrase pair based topic model in step S3, and estimating the topic representation of the bilingual sentence pair or the target field by using a topic vector weighted fusion manner;
[0011] S5: calculating the topic similarity between all candidate bilingual sentence pairs and the target field, and selecting a subset of sentence pairs with the most similar topics to expand the training data set of the field machine translation system.
[0012] In a preferred embodiment of the present application, the distribution pseudo-document of the phrase pair contains context words in two different languages.
[0013] In a preferred embodiment of the present application, the phrase pair based topic model constructed in step S3 is an LDA topic model.
[0014] In a preferred embodiment of the present application, in step S4, the specific steps of estimating the topic representation of the bilingual sentence pair include:
[0015] S401: extracting phrase pairs satisfying word alignment consistency from the candidate bilingual sentence pairs by using a phrase extraction algorithm;
[0016] S402: for each phrase pair, obtaining all contexts thereof in the bilingual parallel corpus, and constructing a distribution pseudo-document;
[0017] S403: weighting the topic vector according to the frequency of the phrase pair in the bilingual sentence pair, and calculating the topic vector of the bilingual sentence pair, which is specifically represented as follows:
[0018]
[0019] V s =<w1(f,e),...w k (f,e)>
[0020] wherein M and N respectively represent the number of source language phrases and target language phrases in the bilingual sentence pair, representing the phrase pair the number of occurrences in bilingual sentence pairs, denotes the phrase pair the value of the k-th dimension in the topic vector, K represents the number of topics, Vs represents the topic vector of the parallel sentence pair s denotes the topic vector of the bilingual sentence pair;
[0021] S404: The importance of the phrase pair in the bilingual sentence pair is measured by using the IDF value, and the topic vector of the phrase pair is weighted by using the IDF importance value, and finally the topic vector of the bilingual sentence pair and the target field development set is calculated, and the specific mathematical expression is as follows:
[0022]
[0023] wherein M, N respectively represent the number of source language phrases and target language phrases in the parallel sentence pair, denotes the importance of the phrase pair, denotes the phrase pair the value of the k-th dimension in the topic vector, K represents the number of topics, Vs represents the topic vector of the parallel sentence pair
[0024] S405: The topic vector calculated above is normalized:
[0025]
[0026] wherein p i (f, e) represents the probability distribution value of the bilingual sentence pair on the i-th topic.
[0027] Further, in step S4, for the phrase pair set extracted in the target field development set, the target field topic vector is calculated in the same way as estimating the topic representation of the bilingual sentence pair.
[0028] In a preferred embodiment of the present application, the specific steps of step S5 include:
[0029] S501: Use Jensen-Shannon divergence as an evaluation index to calculate the similarity between the topic distribution of all candidate bilingual sentence pairs and the topic distribution of the target field development set;
[0030] S502: Sort the candidate bilingual sentence pairs in ascending order according to the Jensen-Shannon divergence score;
[0031] S503: Select the TopN bilingual sentence pairs in the order as the training data of the target field, and build a machine translation system for a specific field, and N is the best value selected according to the translation performance on the development set.
[0032] To solve the above technical problems, the application adopts another technical solution: providing a domain bilingual sentence pair selection system based on topic information, comprising:
[0033] A phrase pair-based topic model training module is configured to build a phrase pair-based topic model, take a phrase pair as a basic unit, and learn a topic representation of the phrase pair by using a context in which the phrase pair is located;
[0034] A bilingual sentence pair and target domain topic vector inference module is configured to infer a topic distribution of a current phrase pair by using the phrase pair-based topic model according to a candidate bilingual sentence pair and a phrase pair set extracted from a target domain, and estimate a topic representation of the bilingual sentence pair or the target domain by using a topic vector weighted fusion manner;
[0035] A topic correlation calculation module is configured to calculate a topic similarity of all candidate bilingual sentence pairs with respect to a target domain, and select a sentence pair subset with the most similar topic to expand a training data set of the domain machine translation system.
[0036] In a preferred embodiment of the application, the phrase pair-based topic model training module comprises:
[0037] A phrase pair extraction unit is configured to perform word alignment processing on all bilingual sentence pairs in a local existing bilingual parallel corpus, and extract phrase pairs satisfying word alignment consistency from the word-aligned bilingual sentence pairs by using a phrase extraction algorithm;
[0038] A phrase pair pseudo-document construction unit is configured to randomly sample part of the phrase pairs, find all bilingual parallel sentence pairs in which the phrase pairs appear in the bilingual corpus, and combine all words in the sentence pairs except the phrase pairs into a document, which is referred to as a distribution pseudo-document of the phrase pair;
[0039] A topic model construction and training unit is configured to build a phrase pair-based topic model, remove stop words, low-frequency words and special characters from the distribution pseudo-document of the phrase pair, and use the pseudo-document as an input of the topic model to train the topic model, obtain a topic distribution of the document, and take the topic distribution of the document as a topic distribution of the corresponding phrase pair.
[0040] In a preferred embodiment of the application, the specific steps of the bilingual sentence pair and target domain topic vector inference module for estimating the topic representation of the bilingual sentence pair comprise:
[0041] S401: Extract phrase pairs satisfying word alignment consistency from the candidate bilingual sentence pairs by using a phrase extraction algorithm;
[0042] S402: For each phrase pair, obtain all contexts of the phrase pair in the bilingual parallel corpus, and construct a distribution pseudo-document;
[0043] S403: weight the theme vector of the phrase pair according to the frequency of the phrase pair in the bilingual sentence pair, and calculate the theme vector of the bilingual sentence pair, which is mathematically expressed as follows:
[0044]
[0045] V s = <w1(f, e),... w k (f, e) > (1)
[0046] wherein M and N respectively represent the number of source language phrases and target language phrases in the bilingual sentence pair, denotes the importance of the phrase pair, denotes the frequency of the phrase pair in the bilingual sentence pair, denotes the phrase pair, denotes the value of the kth dimension in the theme vector, K represents the number of themes, and Vs represents the theme vector of the bilingual sentence pair. s
[0047] S404: measure the importance of the phrase pair in the bilingual sentence pair by using the IDF value, weight the theme vector of the phrase pair by using the IDF importance value, and finally calculate the theme vector of the bilingual sentence pair and the target domain development set, which is mathematically expressed as follows:
[0048]
[0049] wherein M and N respectively represent the number of source language phrases and target language phrases in the parallel sentence pair, denotes the importance of the phrase pair, denotes the phrase pair, denotes the value of the kth dimension in the theme vector, K represents the number of themes, and Vs represents the theme vector of the parallel sentence pair.
[0050] S405: normalize the theme vector calculated above:
[0051]
[0052] wherein p i (f, e) represents the probability distribution value of the bilingual sentence pair on the ith theme.
[0053] In a preferred embodiment of the present application, the theme correlation calculation module calculates the theme similarity of all candidate bilingual sentence pairs to the target domain, and selects a sentence pair subset with the most similar theme to expand the training data set of the domain machine translation system, and the specific steps include:
[0054] S501: calculate the similarity of the theme distribution of all candidate bilingual sentence pairs and the theme distribution of the target domain development set by using Jensen-Shannon divergence as an evaluation index;
[0055] S502: ranking the candidate bilingual sentence pairs in ascending order according to the Jensen-Shannon divergence score;
[0056] S503: selecting the TopN bilingual sentence pairs in the ranking as the training data of the target domain, and building a machine translation system for a specific domain, N being the best value selected according to the translation performance on the development set.
[0057] The present application has the following advantages:
[0058] (1) The present application provides a domain bilingual sentence pair selection method based on topic information, which defines a domain as a collection of texts with different topics mixed, so as to describe the essence of the domain in a more fine-grained manner. Furthermore, the domain relevance of the candidate bilingual sentence pair is evaluated according to the relevance of the topic distribution of the sentence pair to the target domain, so as to realize the automatic selection of the domain bilingual sentence pair.
[0059] (2) The domain bilingual sentence pair selection method based on topic information provided by the present application provides a new method for expanding the training data of a specific domain machine translation system. The main contributions include the following aspects:
[0060] First, a phrase pair-based topic model is proposed, and then the topic representation of the bilingual sentence pair and the target domain is obtained based on the topic distribution of the phrase pair.
[0061] Second, the quality of the bilingual sentence pair is evaluated according to the topic relevance, so as to select the domain-related bilingual sentence pair by using the topic semantic information contained in the text. BRIEF DESCRIPTION OF DRAWINGS
[0062] Figure 1 is a flowchart of the domain bilingual sentence pair selection method based on topic information of the present application;
[0063] Figure 2 is a schematic diagram of the construction process of the distribution pseudo-document of the phrase pair;
[0064] Figure 3 is a structural block diagram of the domain bilingual sentence pair selection system based on topic information. DETAILED DESCRIPTION
[0065] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the scope of protection of the present application can be more clearly defined.
[0066] Please refer to Figure 1 The embodiments of the present application include:
[0067] A domain bilingual sentence pair selection method based on topic information, comprising the following steps:
[0068] S1: based on the existing bilingual parallel corpus, all bilingual sentence pairs are processed by word alignment, and phrase pairs are extracted from the word-aligned bilingual sentence pairs by using a phrase extraction algorithm to meet the consistency of word alignment;
[0069] In this example, the phrase extraction algorithm based on Koehn is used to extract phrase pairs from the word-aligned bilingual parallel corpus. No empty expansion is performed during extraction, and the IDs of the bilingual parallel sentence pairs where the phrase pairs are located are recorded.
[0070] S2: randomly sample part of the phrase pairs, and find all the bilingual parallel sentence pairs where the phrase pairs appear in the bilingual corpus. Specifically, according to the recorded IDs, the context of the current phrase pair is obtained, and all the words in the context except the current phrase pair are combined into a new document, called the distribution pseudo-document of the phrase pair;
[0071] For the sampled phrase pairs, if their frequency of occurrence in the bilingual corpus is greater than a fixed threshold, only part of their context is sampled. Otherwise, the distribution pseudo-document of the phrase pair is constructed using the entire context.
[0072] As shown in Figure 2 , the words that often appear in the context of different phrase pairs can represent the semantics of the phrase pair. For example, the phrase pairs "application|||application" and "application|||application" appear in different bilingual sentence pairs in the corpus. Among them, the phrase pair "application|||application" often appears in the bilingual sentence pairs "development", "database", and other computer-related words. It can be inferred that this phrase pair often appears in computer-related topics. The bilingual sentence pairs where the phrase pair "application|||application" appears often contain words such as "submit", "approval", "approval", and "deadline", which may appear in multiple topics such as education, patents, and job hunting. Therefore, based on the potential topic of the context words of "application|||application", the distribution probability of the phrase pair in each topic can be estimated. Therefore, the present application uses the context words of the phrase pair in the bilingual parallel corpus to construct its pseudo-document, and then obtains the topic distribution of the phrase pair based on the topic distribution of the pseudo-document.
[0073] S3: Construct a topic model based on the phrase pair. After removing stop words, low-frequency words, and special characters from the distribution pseudo-document of the phrase pair, use it as the input of the topic model to train the topic model, obtain the topic distribution of the document, and use the topic distribution of the document as the topic distribution of the corresponding phrase pair. The specific steps include:
[0074] S301: Preprocess the distribution pseudo-document of the phrase pair. Text cleaning is performed on the distribution pseudo-document of each phrase pair, including: removing stop words (such as common high-frequency words such as "of", "is", "in", etc.); removing low-frequency words (setting a word frequency threshold to filter out words with a frequency below the threshold); removing special characters (such as punctuation marks, numbers, non-language symbols, etc.);
[0075] S302: Configure the topic model parameters. Select LDA (Latent Dirichlet Allocation) as the topic model, set the number of topics K, the prior parameters (α, β), the number of iterations, and the convergence threshold, etc.
[0076] S303: Input the preprocessed distribution pseudo-document to the LDA model. The cleaned pseudo-document is converted into a Bag-of-Words representation to generate a document-term matrix.
[0077] S304: Train the LDA topic model. Through Gibbs sampling or variational inference algorithm, iteratively optimize the model parameters, learn the document-topic distribution and topic-term distribution.
[0078] S305: Obtain the topic distribution of the phrase pair. For each phrase pair's distribution pseudo-document, calculate its document-topic probability distribution vector as the topic representation of the phrase pair.
[0079] S306: Store the topic distribution results. Store each phrase pair and its corresponding topic distribution vector for subsequent steps.
[0080] S4: According to the candidate bilingual sentence pair and the target field, develop a set of centrally extracted phrase pairs, use the topic model based on phrase pairs in step S3 to infer the topic distribution of the current phrase pair, and use the topic vector weighted fusion method to estimate the topic representation of the bilingual sentence pair or the target field.
[0081] Taking bilingual sentence pair topic vector calculation as an example, the specific steps are as follows:
[0082] S401: Use the phrase extraction algorithm proposed by Koehn to extract phrase pairs that meet the word alignment consistency from the candidate bilingual sentence pair;
[0083] S402: For each phrase pair, obtain all its contexts in the bilingual parallel corpus to construct a distribution pseudo-document.
[0084] S403: According to the frequency of the phrase pair in the bilingual sentence pair, weight its topic vector, calculate and obtain the topic vector of the bilingual sentence pair, and the specific mathematical representation is as follows:
[0085]
[0086] Vs = <w1(f,e),... w k (f,e)
[0087] where M, N represent the number of source language phrases and target language phrases in bilingual sentence pair respectively, denotes the phrase pair the number of occurrences in bilingual sentence pair, denotes the phrase pair the value of the kth dimension in topic vector, K represents the number of topics, V s denotes the topic vector set of bilingual sentence pair.
[0088] S404: In order to further consider the importance of phrase pair and better obtain the topic vector of bilingual sentence pair and target domain development set, for example, the phrase pair "I|||me" has higher frequency of occurrence in bilingual parallel corpus than the phrase pair "application|||application program", if the topic vector of phrase pair is weighted by the frequency of occurrence of phrase pair, the frequently occurring phrase pair such as "I|||me" will have greater influence on the topic vector of bilingual sentence pair. Therefore, in order to reduce the weight of common phrase pair, the IDF value is also used to measure the importance of phrase pair in bilingual sentence pair, for the phrase pair that frequently occurs in corpus, its IDF value is lower, and its weight is also lower when the topic vector of target domain is finally calculated. The topic vector is weighted by the IDF importance value, and the topic vector of bilingual sentence pair and target domain development set is finally calculated, and the specific mathematical expression is as follows:
[0089]
[0090] V s = <w1(f,e),... w k (f,e)
[0091] where M, N represent the number of source language phrases and target language phrases in parallel sentence pair respectively, the importance of phrase pair obtained by using IDF value, denotes the phrase pair the value of the kth dimension in topic vector, K represents the number of topics, V s denotes the topic vector set of parallel sentence pair.
[0092] S405: In order to ensure that the sum of probability distribution of phrase pair on each topic is l, the topic vector calculated above is further normalized:
[0093]
[0094] where p i (f,e) represents the probability distribution value of bilingual sentence pair on the ith topic, wi (f, e) represents the probability distribution value of the bilingual sentence pair on the i-th topic, p j (f, e) represents the i-th and j-th bilingual pair topic vectors, respectively.
[0095] For the target domain development set, the same method is used to calculate its topic vector, which is not described here.
[0096] S5: Calculate the topic similarity of all candidate bilingual sentence pairs and the target domain, and select the subset of the most similar sentence pairs to expand the training data set of the domain machine translation system. The specific steps include:
[0097] S501: Use Jensen-Shannon divergence as an evaluation index to calculate the similarity of the topic distribution of all candidate bilingual sentence pairs and the topic distribution of the target domain development set;
[0098] S502: Sort the candidate bilingual sentence pairs in ascending order according to the Jensen-Shannon divergence score, and the lower the score, the higher the relevance of the parallel sentence pairs to the domain;
[0099] Jensen-Shannon divergence is an index for calculating the similarity of two probability distributions in statistics, and its mathematical definition is as follows:
[0100]
[0101] Where, p i (f, e) represents the probability distribution value of the bilingual sentence pair on the i-th topic, p i (dev) represents the probability distribution value of the target domain development set on the i-th topic.
[0102] S503: Select the TopN bilingual sentence pairs in order as the training data of the target domain to build a machine translation system for a specific domain, N is the best value selected according to the translation performance on the development set, and in this example, TopN = {50K, 200K, 400K, 600K, 800K, 1,000K}.
[0103] Referring to Figure 3 In the example of the present application, a domain bilingual sentence pair selection system based on topic information is also provided, which includes a phrase pair-based topic model training module, a bilingual sentence pair and target domain topic vector inference module, and a topic relevance calculation module.
[0104] The phrase pair-based topic model training module is used to construct a phrase pair-based topic model, taking phrase pairs as the basic unit, and learning their topic representation using the context in which they are located.
[0105] The bilingual sentence pair and target field topic vector inference module is configured to infer the topic distribution of the current phrase pair by using the phrase pair-based topic model according to the candidate bilingual sentence pair and the set of extracted phrase pairs in the target field development set, and estimate the topic representation of the bilingual sentence pair or the target field by using a topic vector weighted fusion manner.
[0106] The topic correlation calculation module is configured to calculate the topic similarity of all candidate bilingual sentence pairs with the target field, and select a sentence pair subset with the most similar topic to expand the training data set of the field machine translation system.
[0107] Further, the phrase pair-based topic model training module comprises a phrase pair extraction unit, a phrase pair pseudo document construction unit, and a topic model construction and training unit.
[0108] The phrase pair extraction unit is configured to perform word alignment processing on all bilingual sentence pairs in the existing bilingual parallel corpus, and extract phrase pairs satisfying word alignment consistency from the word-aligned bilingual sentence pairs by using a phrase extraction algorithm.
[0109] The phrase pair pseudo document construction unit is configured to randomly sample part of the phrase pairs, find all bilingual parallel sentence pairs in which the phrase pairs appear in the bilingual corpus, and combine all words in the sentence pairs except the phrase pairs into a document, which is referred to as a distribution pseudo document of the phrase pair.
[0110] The topic model construction and training unit is configured to construct a phrase pair-based topic model, remove stop words, low-frequency words and special characters from the distribution pseudo document of the phrase pair, and use the document as an input of the topic model to train the topic model, obtain the topic distribution of the document, and use the topic distribution of the document as the topic distribution of the corresponding phrase pair.
[0111] The field bilingual sentence pair selection system based on topic information according to the present example can execute the field bilingual sentence pair selection method based on topic information provided by the present application, and can execute any combination of the steps of the method example, and has the corresponding functions and beneficial effects of the method.
[0112] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.
Claims
1. A domain bilingual sentence pair selection method based on topic information, characterized in that, The method comprises the following steps: S1: based on the existing bilingual parallel corpus, all bilingual sentence pairs are processed by word alignment, and phrase pairs satisfying word alignment consistency are extracted from the word-aligned bilingual sentence pairs by using a phrase extraction algorithm; S2: a part of the phrase pairs are randomly sampled, and all bilingual parallel sentence pairs in which the phrase pairs appear are found in the bilingual corpus, all words in the sentence pairs except the phrase pairs are combined into a document, referred to as a distribution pseudo-document of the phrase pair; S3: a phrase pair-based topic model is constructed, the distribution pseudo-document of the phrase pair is used as the input of the topic model after removing stop words, low-frequency words and special characters, the topic model is trained, the topic distribution of the document is obtained, and the topic distribution of the document is used as the topic distribution of the corresponding phrase pair; S4: according to the candidate bilingual sentence pair and the target field development extracted phrase pair set, the topic distribution of the current phrase pair is inferred by using the phrase pair-based topic model in step S3, and the topic representation of the bilingual sentence pair or the target field is estimated by using the topic vector weighted fusion mode; S5: the topic similarity of all candidate bilingual sentence pairs and the target field is calculated, and the sentence pair subset with the most similar topic is selected to expand the training data set of the field machine translation system.
2. The method of claim 1, wherein the selecting of the bilingual sentence pair based on the theme information comprises: determining a theme of the input sentence; and selecting the bilingual sentence pair based on the theme. The distribution pseudo-document of the phrase pair contains context words in two different languages.
3. The method of claim 1, wherein the selecting of the bilingual sentence pair based on the theme information comprises: selecting a bilingual sentence pair based on the theme information. The phrase pair-based topic model constructed in step S3 is an LDA topic model.
4. The method of claim 1, wherein the selecting of the bilingual sentence pair based on the theme information comprises: selecting a bilingual sentence pair based on the theme information. In step S4, the specific steps of estimating the topic representation of the bilingual sentence pair include: S401: using a phrase extraction algorithm to extract phrase pairs satisfying word alignment consistency from the candidate bilingual sentence pair; S402: for each phrase pair, all contexts in the bilingual parallel corpus are obtained, and a distribution pseudo-document is constructed; S403: the topic vector of the bilingual sentence pair is calculated by weighting the topic vector according to the frequency of the phrase pair in the bilingual sentence pair, and the specific mathematical expression is as follows: V s = <w1(f, e),... w k (f, e)> where M, N represent the number of source language phrases and target language phrases in the bilingual sentence pair respectively, denotes a phrase pair occurs in the bilingual sentence pair, denotes a phrase pair the value of the kth dimension in the topic vector, K represents the number of topics, Vs represents the topic vector of the bilingual sentence pair; S404: the importance of the phrase pair in the bilingual sentence pair is measured by using the IDF value, and the topic vector is weighted by using the IDF importance value, and finally the topic vector of the bilingual sentence pair and the target field development set is calculated, and the specific mathematical expression is as follows: where M, N represent the number of source language phrases and target language phrases in parallel sentence pairs respectively, denotes the importance of a phrase pair, denotes the phrase pair the value of the kth dimension in the topic vector, K represents the number of topics, Vs represents the topic vector of parallel sentence pairs; S405: the topic vector calculated above is normalized: where p i (f, e) denotes the probability distribution value of the bilingual sentence pair on the i-th topic. 5.The domain bilingual sentence pair selection method based on theme information according to claim 4, characterized in that, In step S4, for the phrase pair set extracted in the target field development, the target field topic vector is calculated in the same way as estimating the topic representation of the bilingual sentence pair. 6.The domain bilingual sentence pair selection method based on theme information according to claim 1, wherein, The specific steps of step S5 include: S501: using Jensen-Shannon divergence as an evaluation index, the similarity of the topic distribution of all candidate bilingual sentence pairs and the topic distribution of the target field development set is calculated; S502: the candidate bilingual sentence pairs are sorted in ascending order according to the Jensen-Shannon divergence score; S503: the bilingual sentence pairs in the top N are selected as the training data of the target field, a machine translation system for a specific field is built, and N is the best value selected according to the translation performance on the development set.
7. A domain bilingual sentence pair selection system based on topic information, characterized by, The method comprises the following steps: The phrase pair-based topic model training module is configured to construct a phrase pair-based topic model, and learn a topic representation of a phrase pair by using a context in which the phrase pair is located. The bilingual sentence pair and target domain topic vector inference module is configured to infer a topic distribution of a current phrase pair by using the phrase pair-based topic model according to a candidate bilingual sentence pair and a target domain development set, and estimate a topic representation of the bilingual sentence pair or the target domain by using a topic vector weighted fusion manner. The topic relevance calculation module is configured to calculate a topic similarity of all candidate bilingual sentence pairs and the target domain, and select a sentence pair subset with the most similar topic to expand a training data set of the domain machine translation system. 8.The domain bilingual sentence pair selection system based on theme information according to claim 7, characterized in that, The phrase pair-based topic model training module comprises: The phrase pair extraction unit is configured to perform word alignment processing on all bilingual sentence pairs in a locally existing bilingual parallel corpus, and extract phrase pairs satisfying word alignment consistency from the word-aligned bilingual sentence pairs by using a phrase extraction algorithm. The phrase pair pseudo document construction unit is configured to randomly sample part of the phrase pairs, find all bilingual parallel sentence pairs in which the phrase pairs appear in the bilingual corpus, and combine all words in the sentence pairs except the phrase pairs into a document, which is referred to as a distribution pseudo document of the phrase pair. The topic model construction and training unit is configured to construct a phrase pair-based topic model, use the distribution pseudo document of the phrase pair after removing stop words, low-frequency words and special characters as an input of the topic model, train the topic model, obtain a topic distribution of the document, and use the topic distribution of the document as a topic distribution of the corresponding phrase pair. 9.The domain bilingual sentence pair selection system based on theme information according to claim 7, wherein, The specific steps of estimating the topic representation of the bilingual sentence pair by the bilingual sentence pair and target domain topic vector inference module comprise: S401: Extracting phrase pairs satisfying word alignment consistency from the candidate bilingual sentence pairs by using a phrase extraction algorithm. S402: For each phrase pair, obtaining all contexts of the phrase pair in the bilingual parallel corpus, and constructing a distribution pseudo document. S403: Weighting the topic vector of the phrase pair according to a frequency of the phrase pair in the bilingual sentence pair, and calculating the topic vector of the bilingual sentence pair, which is specifically represented as follows: V s = <w1(f, e),... w k (f, e) > where M, N represent the number of source language phrases and target language phrases in the bilingual sentence pair respectively, denotes a phrase pair occurs in the bilingual sentence pair, denotes a phrase pair the value of the kth dimension in the topic vector, K represents the number of topics, V s denotes the topic vector of the bilingual sentence pair; S404: Measuring the importance of the phrase pair in the bilingual sentence pair by using an IDF value, weighting the topic vector of the phrase pair by using the IDF importance value, and finally calculating the topic vector of the bilingual sentence pair and the target domain development set, which is specifically represented as follows: where M, N represent the number of source language phrases and target language phrases in parallel sentence pairs respectively, denotes the importance of a phrase pair, denotes the phrase pair the value of the k-th dimension in the topic vector, K represents the number of topics, V s denotes the topic vector of parallel sentence pairs; S405: Normalizing the calculated topic vector. where p i (f, e) denotes the probability distribution value of the bilingual sentence pair on the i-th topic. 10.The domain bilingual sentence pair selection system based on theme information according to claim 7, characterized in that, The specific steps of calculating the topic similarity of all candidate bilingual sentence pairs and the target domain by the topic relevance calculation module, and selecting a sentence pair subset with the most similar topic to expand the training data set of the domain machine translation system comprise: S501: Calculating the similarity of the topic distribution of all candidate bilingual sentence pairs and the topic distribution of the target domain development set by using Jensen-Shannon divergence as an evaluation index. S502: Sorting the candidate bilingual sentence pairs in ascending order according to the Jensen-Shannon divergence score. S503: Select the top N bilingual sentence pairs in the order as the training data of the target field, and build a machine translation system for a specific field, N is the best value selected according to the translation performance on the development set.