A cross-domain method for measuring the comparability of bilingual texts

Abstract

The present invention particularly relates to a cross-domain bilingual text comparability quantification method, comprising the following steps: S100, pre-processing the existing comparable corpus by Chinese word segmentation, English part-of-speech restoration, stop word filtering, etc., and training a bilingual topic model; S200, predicting the topic distribution of source language texts and target language texts, and calculating the relevance of texts in different languages; S300, calculating the relevance of different language vocabularies through the topic distribution of different language vocabularies, expanding the adaptability of bilingual knowledge in different fields, and using vocabulary to calculate the relevance of texts in different languages; S400, quantifying the comparability of texts in different languages ​​by using the relevance of text topics and vocabularies in different languages. Calculate the relevance of different language vocabularies through bilingual topics, expand domain bilingual knowledge, improve the adaptability of text comparability quantification to different fields, and enhance the effect of text comparability quantification.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-language information processing, and in particular to a cross-domain bilingual text comparability quantification method. Background Art

[0002] Bilingual text comparable corpus has been widely used for mining fine-grained translation equivalence pairs such as parallel sentence pairs, bilingual multi-words and mutual translation vocabulary due to its advantages such as wide sources, strong timeliness and balanced fields, and has become a basic resource for statistical machine translation and cross-language information retrieval. At the same time, it has also been used for cross-language information processing such as cross-language document clustering, multilingual distribution system class label description, cross-language sentiment analysis and cross-language text plagiarism monitoring.

[0003] The comparability of bilingual texts has a great impact on the extraction of fine-grained resources. The higher the comparability, the more obvious the effect of extracting fine-grained mutual translation equivalent pairs. When a high-comparability subset is separated from a comparable corpus, although the size of the corpus collection is reduced, the extracted mutual translation vocabulary is more accurate. The emergence of the value of comparability has shifted people's evaluation of the quality of comparable corpora from qualitative classification to quantitative research. Comparability quantification currently uses lexical-level information such as word frequency sequences, lexical translation expectations, lexical overlap, keywords, and machine translation for matching, which is difficult to reflect the relevance of the main structural information of bilingual texts. In addition, the use of comparability quantification alone loses lexical-level information, and the differences in existing lexical knowledge in the corpus field limit the performance of quantification. Therefore, comparability quantification work still needs further development. Summary of the invention

[0004] The purpose of the present invention is to provide a cross-domain bilingual text comparable quantification method with good quantification effect and strong adaptability.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a cross-domain bilingual text comparability quantification method, comprising the following steps: S100, pre-processing the existing comparable corpus such as Chinese word segmentation, English part-of-speech restoration, and stop word filtering, and training a bilingual topic model; S200, predicting the topic distribution of source language texts and target language texts, and calculating the relevance of texts in different languages; S300, calculating the relevance of vocabularies in different languages ​​through the topic distribution of vocabularies in different languages, expanding the adaptability of bilingual knowledge in different fields, and calculating the relevance of texts in different languages ​​using vocabularies; S400, quantifying the comparability of texts in different languages ​​using the relevance of text topics and vocabularies in different languages.

[0006] Compared with the prior art, the present invention has the following technical effects: calculating the relevance of different language vocabulary through bilingual topics, expanding domain bilingual knowledge, improving the adaptability of text comparability quantification to different fields, and enhancing the effect of text comparability quantification. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0008] Combine the following Figure 1 , the present invention is further described in detail.

[0009] See also Figure 1 , a cross-domain bilingual text comparability quantification method, including the following steps: S100, pre-processing the existing comparable corpus such as Chinese word segmentation, English part-of-speech restoration, and stop word filtering, and training the bilingual topic model; S200, predicting the topic distribution of the source language text and the target language text, and calculating the relevance of texts in different languages; S300, calculating the relevance of different language words through the topic distribution of different language words, expanding the adaptability of bilingual dictionaries in different fields, and calculating the relevance of texts in different languages ​​through the relevance of words; S400, using the relevance of text topics and words in different languages, calculating the quantification of the comparability of texts in different languages. Calculate the relevance of different language words through bilingual topics, expand the field bilingual knowledge, improve the adaptability of text comparability quantification to different fields, and enhance the effect of text comparability quantification.

[0010] Furthermore, in step S200, a bilingual topic model is used to predict the source language chapter and target language passages The topic distribution probabilities are and , calculate the cosine similarity of topic distribution to measure the relevance of different language chapters .

[0011]

[0012] in, is the cross-language topic vector Middle ( Themes, is the number of cross-language topics.

[0013] Furthermore, in step S300, by analyzing the topic distribution of different language words, the correlation of different language words is calculated, the adaptability of the bilingual dictionary in different fields is expanded, and the correlation of different language chapters is calculated through the correlation of words. The following steps are included: in S310, the topic distribution of source language words and target language words is predicted and obtained; in S320, the correlation of different language words is calculated using the topic distribution of words; in S330, the correlation of different language words is calculated through the correlation of words using the correlation of different language words.

[0014] Furthermore, the step S310 includes the following steps: using a cross-language topic model to obtain source language vocabulary and target language vocabulary On topic The probability of the following is and .

[0015] According to the Bayesian formula, the source language vocabulary Theme The probability is It can be calculated according to the following formula:

[0016]

[0017] According to the total probability formula, the probability of occurrence of source language vocabulary is It can be expressed as:

[0018]

[0019] Prior probability It obeys uniform distribution, that is, each topic occurs with equal probability under non-specific documents, that is, ; Source language vocabulary The probability of occurrence under all topics is constant, let ,therefore:

[0020]

[0021] Similarly, the target language vocabulary Theme The probability is for:

[0022]

[0023] Furthermore, in step S320, when the source language vocabulary and target language vocabulary When translating word pairs, define the relevance of the words =1; when and For non-inter-translation word pairs, the cosine similarity of the topic is used to measure and The correlation is:

[0024]

[0025] Furthermore, in step S330, different language chapters are calculated. Chinese Vocabulary and Relevance:

[0026]

[0027] in, For Documentation The number of words in For Documentation The number of words in the word list, taking the parameter .

[0028] Counting different language chapters Relevance:

[0029]

[0030] Furthermore, in step S400, the correlation between the topics and words of different language passages is used to calculate the quantitative comparability of different language passages. .

[0031]

[0032] Among them, take the parameter .

Claims

1. A cross-domain bilingual text comparability quantification method, characterized by: The steps include: S100: Perform Chinese word segmentation, English part-of-speech restoration, and stop word filtering preprocessing on the existing comparable corpus, and train a bilingual topic model; S200, predict the topic distribution of source language chapters and target language chapters, and calculate the correlation between different language chapters; S300, by subject distribution of different language words, calculate the correlation of different language words, expand the adaptability of bilingual dictionaries in different fields, and calculate the correlation of different language chapters by the correlation of words; S400, using the correlation between the themes and vocabulary of different language texts, calculate the quantification of the comparability of different language texts; The step S300 includes the following steps: In S310, the topic distribution of the source language vocabulary and the target language vocabulary is predicted and obtained; In S320, the correlation of words in different languages ​​is calculated using the topic distribution of the words; In S330, the correlation of words in different languages ​​is used to calculate the correlation of passages in different languages ​​through the correlation of words; In step S320, when the source language vocabulary and target language vocabulary When translating word pairs, define the relevance of the words when and For non-inter-translation word pairs, the cosine similarity of the topic is used to measure and Relevance: In step S330, the following formula is used to calculate the different language chapters: and Chinese Vocabulary and Relevance: Among them, the parameters In step S330, different language chapters are calculated according to the following formula: and Relevance: in, For Documentation The number of words in For Documentation The number of words in the text.

2. A cross-domain bilingual text comparability quantification method as claimed in claim 1, characterized in that: In step S200, a bilingual topic model is used to predict the source language chapter and target language passages The topic distribution probabilities are and The cosine similarity of topic distribution is calculated according to the following formula to measure the relevance of different language chapters. Among them, Z k is the kth topic in the cross-lingual topic vector Z, 1≤k≤K, and K is the number of the cross-lingual topics.

3. A cross-domain bilingual text comparability quantification method as claimed in claim 1, characterized in that: The step S310 includes the following steps: Using cross-language topic models to obtain source language vocabulary and target language vocabulary In Topic Z k The probability of the following is and According to the Bayesian formula, the source language vocabulary Theme Z k The probability is It can be calculated according to the following formula: According to the total probability formula, the probability of occurrence of source language vocabulary is It can be expressed as: The prior probability P(Z) follows a uniform distribution, that is, each topic occurs with equal probability in a non-specific document, that is, P(Z) = P(Z k ); source language vocabulary The probability of occurrence under all topics is constant, let therefore: make Similarly, the target language vocabulary Theme Z k The probability is for:

4. A cross-domain bilingual text comparability quantification method as claimed in claim 1, characterized in that: In step S400, the correlation between the topics and words of different language passages is used to calculate the quantitative comparability of different language passages.

5. A cross-domain bilingual text comparability quantification method as claimed in claim 4, characterized in that: Quantification of the comparability of texts in different languages Calculated as follows: Among them, μ is the weight coefficient.

6. A cross-domain bilingual text comparability quantification method as claimed in claim 5, characterized in that: The parameter μ=0.3.

Images