A method for constructing a Chinese-English bilingual dictionary based on academic literature

Abstract

The invention discloses a method for constructing a Chinese-English bilingual dictionary based on academic documents, including: collecting Chinese and English title corpora, Chinese and English keyword corpora and Chinese and English annotation corpora in the academic documents, and preprocessing the corpora; constructing Chinese and English title translation thesaurus, Chinese and English keyword translation thesaurus and Chinese and English annotation translation thesaurus respectively; optimizing the Chinese and English keyword translation thesaurus and the Chinese and English annotation translation thesaurus respectively; extracting Chinese vocabulary in the optimized Chinese and English keyword translation thesaurus and the Chinese and English annotation translation thesaurus, and constructing a Chinese vocabulary list; optimizing the Chinese and English title translation thesaurus; translating the Chinese vocabulary list through the Chinese and English title translation thesaurus, the Chinese and English keyword translation thesaurus and the Chinese and English annotation translation thesaurus, and constructing a Chinese and English bilingual dictionary. The invention greatly improves the accuracy of the Chinese and English bilingual dictionary, and solves the problem that the synonym processing in the existing Chinese and English bilingual dictionary construction method is not effective.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method for constructing a Chinese-English bilingual dictionary based on academic literature. Background Art

[0002] The construction of bilingual dictionaries is the basis of machine translation, and its accuracy greatly affects the results of machine translation. Currently, the main methods used to build bilingual dictionaries include those based on parallel corpora, comparable corpora, and seed dictionaries.

[0003] The method of building a bilingual dictionary based on comparable corpus is mainly based on word frequency statistics, extracting high-frequency words from text corpus of each monolingual language type, and using each high-frequency word and its corresponding representation result to build a bilingual dictionary. Alternatively, based on TF-IDF statistics, each monolingual feature word is used to build feature word pairs, and feature word pairs with high similarity are screened out to build a bilingual dictionary. Comparable corpus is easy to obtain but has low alignment. Building a bilingual dictionary based on comparable corpus is not only difficult and has low accuracy, but also has poor processing effect on synonyms. Building a bilingual dictionary based on a seed dictionary is greatly affected by the scale and quality of the seed dictionary and consumes a lot of manpower and time.

[0004] Parallel corpora have high-quality alignment information. Building a bilingual dictionary based on parallel corpora is not only accurate but also easy to implement. However, parallel corpus resources are scarce and difficult to build. Most of the existing parallel corpora are constructed based on the similarity of sentences in two different languages. The corpus alignment is poor, and the accuracy of the bilingual dictionary built based on this parallel corpus is low. Academic literature is a corpus that has been manually reviewed many times and can provide high-quality parallel corpus resources. Extracting parallel corpora based on academic literature, building and optimizing bilingual dictionaries based on academic literature parallel corpora can effectively improve the accuracy of bilingual dictionaries and effectively solve the problem of poor synonym processing in existing dictionaries, which is of great significance to machine translation research. Summary of the invention

[0005] To solve the above technical problems, the purpose of the present invention is to provide a method for constructing a Chinese-English bilingual dictionary based on academic literature, which is used to solve the problems of low accuracy and poor synonym processing in existing Chinese-English bilingual dictionaries.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A method for constructing a Chinese-English bilingual dictionary based on academic literature, comprising:

[0008] Step A collects Chinese and English title corpus, Chinese and English keyword corpus and Chinese and English annotation corpus in the Chinese full text of academic literature, and pre-processes the corpus;

[0009] Step B constructs Chinese and English title translation word libraries, Chinese and English keyword translation word libraries, and Chinese and English annotation translation word libraries respectively;

[0010] Step C optimizes the Chinese and English keyword translation word libraries and the Chinese and English annotation translation word libraries respectively;

[0011] Step D extracts Chinese words from the optimized Chinese-English keyword translation word library and the Chinese-English annotation translation word library, and constructs a Chinese word list;

[0012] Step E: Optimizing the Chinese and English title translation word library;

[0013] Step F translates the Chinese word list through the Chinese-English title translation word library, the Chinese-English keyword translation word library, and the Chinese-English annotation translation word library to construct a Chinese-English bilingual dictionary.

[0014] Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

[0015] The present invention constructs a Chinese-English bilingual dictionary based on academic literature, ensuring the alignment quality and accuracy of the original corpus. A Chinese-English title translation vocabulary, a Chinese-English keyword translation vocabulary, and a Chinese-English annotation translation vocabulary are constructed, and the translation vocabulary is optimized to further improve the accuracy of the dictionary. The Chinese vocabulary is extracted from the translation vocabulary and translated into three translation vocabulary, and the translation results are integrated and deduplicated to construct a Chinese-English bilingual dictionary, retaining the synonyms of the bilingual dictionary. The research on this problem is of great significance to machine translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method of constructing a Chinese-English bilingual dictionary based on academic literature;

[0017] Figure 2 It is a flowchart for constructing and optimizing the Chinese and English keyword translation word library;

[0018] Figure 3 It is a flowchart for constructing and optimizing the Chinese-English annotated translation vocabulary;

[0019] Figure 4 It is a flow chart for constructing and optimizing the Chinese-English title translation vocabulary. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with embodiments and drawings.

[0021] like Figure 1 As shown in the figure, the process of constructing a Chinese-English bilingual dictionary based on academic literature includes the following steps:

[0022] Step A collects Chinese and English title corpus, Chinese and English keyword corpus and Chinese and English annotation corpus in the Chinese full text (such as Artificial Intelligence (AI)) of academic literature, and preprocesses the corpus;

[0023] Step B constructs Chinese and English title translation word libraries, Chinese and English keyword translation word libraries, and Chinese and English annotation translation word libraries respectively;

[0024] Step C optimizes the Chinese and English keyword translation word libraries and the Chinese and English annotation translation word libraries respectively;

[0025] Step D extracts Chinese words from the optimized Chinese-English keyword translation word library and the Chinese-English annotation translation word library, and constructs a Chinese word list;

[0026] Step E: Optimizing the Chinese and English title translation word library;

[0027] Step F translates the Chinese word list through the Chinese-English title translation word library, the Chinese-English keyword translation word library, and the Chinese-English annotation translation word library to construct a Chinese-English bilingual dictionary.

[0028] The above step A specifically includes the following steps:

[0029] Step A1 extracts Chinese and English titles, Chinese and English keywords, and Chinese and English annotation corpora of academic documents; the Chinese and English annotation parts of the academic documents are extracted by three different automatic extraction algorithms and marked as Chinese, NChinese, and MChinese, hereinafter referred to as C, N, and M; examples of the data structures of the three different automatic extraction algorithms are shown in Table 1, and some irrelevant columns are not shown;

[0030] Step A2 performs word segmentation on the corpus; the Chinese title is segmented, and the English title is segmented by spaces;

[0031] Step A3: Filter illegal corpus: Count the number of Chinese and English keywords in each document, and remove keyword materials with inconsistent numbers of Chinese and English keywords.

[0032] Table 1

[0033]

[0034] The above step B specifically includes:

[0035] Step B1: Constructing a Chinese-English title translation word library: aligning the Chinese-English title corpus to construct key-value word pairs, and constructing a Chinese-English title translation word library according to the bilingual comparison relationship;

[0036] Step B2: Constructing a Chinese-English keyword translation word library: aligning the Chinese-English keyword corpus with the sequential alignment method to construct key-value word pairs, calculating the similarity of the word pairs, and constructing a Chinese-English keyword translation word library with word pairs having a similarity greater than k (k∈(0.5,1));

[0037] Step B3: Construction of Chinese-English annotation translation word library: Analyze and clean the Chinese-English annotation corpus obtained by three different automatic extraction algorithms, summarize all word pairs, and form a key-value word pair of one Chinese word and multiple English words, so as to obtain a Chinese-English annotation translation word library.

[0038] The corpus data cleaning rules in step B3 above are:

[0039] 1) When the three translation results are consistent, N is used as the Chinese translation to form a word pair.

[0040] 2) When any two translation results are consistent, N or M is used as the Chinese translation to form a word pair.

[0041] 3) When the three translation results are different, but N or M contains the original English text and brackets, the result without the English and brackets will be used as the Chinese translation to form a word pair.

[0042] 4) When the three translation results are different and the English words are all capitalized, no word pair is formed.

[0043] 5) When the three translation results are all different, but N or M is a subset of the other two translation results, the subset is used as the Chinese translation to form a word pair.

[0044] 6) When the three translation results are different and have no mutual inclusion relationship, similarity calculation is performed, and the similarities are arranged in descending order. The translation result with the highest similarity greater than n (n∈(0.5,1)) is selected to form a word pair.

[0045] The above step C specifically includes the following steps:

[0046] Step C1: Optimize the Chinese and English keyword translation word library: Analyze the errors in the Chinese and English keyword translation word library, and use different methods to optimize according to different error categories. The construction and optimization process of the optimized Chinese and English keyword translation word library is as follows: Figure 2 As shown;

[0047] Step C2: Optimize the Chinese and English annotation translation word library: adjust the data cleaning method of the Chinese and English annotation translation word library to improve the accuracy of the translation word library; the construction and optimization flow chart of the optimized Chinese and English annotation translation word library is as follows: Figure 3 shown.

[0048] In the above step C1, the alignment order of Chinese and English keywords is incorrect, and the secondary alignment method is used for optimization. Each Chinese and English keyword in a document is composed into word pairs, and the word frequencies of the entries in the Chinese and English keyword translation word library under different alignment orders are counted, and the word pair combination with the largest sum of all keyword word frequencies is selected as a reasonable alignment scheme. Synonyms are effectively retained based on the total frequency of word pair combinations rather than the frequency of word pairs.

[0049] For typos or spelling errors, we use a filtering algorithm based on an existing vocabulary to optimize. When a keyword has a typo or spelling error, we use Wikipedia to determine whether it is in the collection. If it is not in the collection and the word frequency is less than the set threshold, it will be deleted from the translation vocabulary.

[0050] In view of the weak correlation between Chinese and English keywords, the filtering algorithm of the language similarity model is used to correct the dictionary. The similarity of Chinese and English keywords is calculated to filter out the word pairs with weak correlation between Chinese and English.

[0051] In the above step C2, the data cleaning rules are adjusted as follows:

[0052] 1) When the three translation results are different and the English is all capitalized, similarity calculation is performed and the translation result with the highest similarity is selected;

[0053] 2) When the three translation results are different, but N or M is the superset of another translation result, the superset is used as the Chinese translation to form a word pair;

[0054] 3) When the three translation results are different and do not contain each other, they are sorted in descending order of similarity, and the translation result with the highest similarity greater than m (m∈(0.5,1)) is selected to form a word pair;

[0055] 4) Traverse the word pairs in the key-value dictionary. If the Chinese does not contain commas, split the English translation with the separator and traverse the segmentation results. Exclude segmentation results with a length less than 3; exclude segmentation results with all numbers; if a segmentation result ends with a number, cut off the number part, if the non-number part is a subset of other segmentation results or is the same as other segmentation results, exclude this segmentation result; if a segmentation result contains spaces, after removing the spaces, it is a subset of other segmentation results or is equal to other segmentation results, exclude this segmentation result;

[0056] 5) Calculate the similarity of the key-value word pairs and retain the i (i ≥ 5) English translations with the highest similarity.

[0057] The above step D specifically includes: extracting the key values ​​of the optimized Chinese-English keyword translation word library and the Chinese-English annotation translation word library to obtain Chinese vocabulary, integrating and removing duplicates of the extracted Chinese vocabulary, and constructing a Chinese vocabulary list.

[0058] The above step E specifically includes: traversing the words in the Chinese vocabulary, and retrieving the translation results in the Chinese and English title translation library. Perform word frequency statistics on all translation results, and filter out low-frequency word pairs. On this basis, the disorder problem is dealt with, and the translation results with the same lowercase but different orders are considered to be the same translation result and integrated, and the word frequencies are added. Further, the inclusion relationship is processed, the frequency of translation results with inclusion relationship is added and the original frequency ratio is calculated, and finally the final translation result is obtained by combining the similarity, frequency and original frequency ratio. The word pairs obtained through the title translation library are summarized, and the key-value dictionary of the Chinese and English title translation library is constructed for query matching. The Chinese and English title translation library construction and optimization process is as follows: Figure 4 shown.

[0059] The above step F specifically includes: traversing the Chinese words in the Chinese word list, finding the corresponding value from the key-value of the three translation libraries of Chinese and English titles, Chinese and English keywords, and Chinese and English annotations, integrating and removing duplicates of these translation results to form synonym pairs of the Chinese dictionary, and entering the database to build a Chinese-English bilingual dictionary.

[0060] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A method for constructing a Chinese-English bilingual dictionary based on academic literature, characterized in that: The method comprises the following steps: Step A, collect the Chinese and English title corpus, Chinese and English keyword corpus and Chinese and English annotation corpus in the Chinese full text of academic literature, and pre-process the corpus; The step A specifically includes the following steps: Step A1, extracting Chinese and English titles, Chinese and English keywords, and Chinese and English annotation corpora of academic documents: wherein the Chinese and English annotation corpora of academic documents are extracted by three different automatic extraction algorithms, which are marked as Chinese, NChinese, and MChinese, hereinafter referred to as C, N, and M; Step A2, segment the corpus: segment the Chinese title by words, and segment the English title by spaces; Step A3, filtering illegal corpus: counting the number of Chinese and English keywords in each document, and removing keyword materials with inconsistent numbers of Chinese and English keywords; Step B, respectively constructing Chinese and English title translation word libraries, Chinese and English keyword translation word libraries, and Chinese and English annotation translation word libraries; The step B specifically includes: step B1, constructing a Chinese-English title translation word library: aligning Chinese and English title corpora to construct key-value word pairs, and constructing a Chinese-English title translation word library according to the bilingual comparison relationship; Step B2, constructing a Chinese-English keyword translation word library: aligning the Chinese-English keyword corpus with the sequential alignment method to construct key-value word pairs, calculating the similarity of the word pairs, and constructing a Chinese-English keyword translation word library with word pairs with similarity higher than k, k∈(0.5,1); Step B3, constructing a Chinese-English annotation translation word library: analyzing and cleaning the Chinese-English annotation corpus obtained by three different automatic extraction algorithms, summarizing all word pairs to form a key-value word pair of one Chinese word and multiple English words, thereby obtaining a Chinese-English annotation translation word library; The corpus cleaning rules in step B3 are as follows: 1) When the three translation results are consistent, N is used as the Chinese translation to form a word pair; 2) When any two translation results are consistent, N or M is used as the Chinese translation to form a word pair; 3) When the three translation results are different, but N or M contains the original English text and brackets, the result without the English and brackets is used as the Chinese translation to form a word pair; 4) When the three translation results are different and the English is all capitalized, no word pair is formed; 5) When the three translation results are different, but N or M is a subset of the other two translation results, the subset is used as the Chinese translation to form a word pair; 6) When the three translation results are different and there is no mutual inclusion relationship, similarity calculation is performed, the similarities are arranged in descending order, and the translation result with the highest similarity and greater than n is taken to form a word pair, n∈(0.5,1); Step C, respectively optimizing the Chinese and English keyword translation word library and the Chinese and English annotation translation word library; The step C specifically includes the following steps: step C1, optimizing the Chinese-English keyword translation word library: analyzing the errors existing in the Chinese-English keyword translation word library, and optimizing it by using different methods according to different error categories; for errors in the alignment order of Chinese-English keywords, optimizing it by using the secondary alignment method, forming each Chinese-English keyword in a document into a word pair, counting the word frequency of the word pair in the Chinese-English keyword translation word library under different alignment orders, and selecting the word pair combination with the largest sum of all keyword word frequencies as a reasonable alignment method; for typos or spelling errors, optimizing it by using a filtering algorithm based on an existing word list, when a keyword has a typo or spelling error, using Wikipedia to determine whether it is in the collection, if it is not in the collection and the word frequency is less than a set threshold, it will be deleted from the Chinese-English keyword translation word library; for weak correlation between Chinese and English keywords, using the filtering algorithm of the language similarity model to correct the word pairs, perform Chinese-English keyword similarity calculation, and filter out word pairs with weak correlation between Chinese and English; Step C2, Chinese and English annotation translation word library optimization: adjust the corpus cleaning method of the Chinese and English annotation translation word library to improve the accuracy of the translation word library; In step C2, the data cleaning rules are adjusted as follows: 1) When the three translation results are different and the English is all capitalized, similarity calculation is performed and the translation result with the highest similarity is selected; 2) When the three translation results are different, but N or M is the parent set of another translation result, the parent set is used as the Chinese translation to form a word pair; 3) When the three translation results are different and there is no mutual inclusion relationship, they are arranged in descending order of similarity, and the translation result with the highest similarity greater than m is selected to form a word pair, where m∈(0.5,1); 4) The word pairs in the key-value dictionary are traversed, and if the Chinese does not contain Comma, split the English translation with the separator, traverse the segmentation results, exclude the segmentation results with a length less than 3, exclude the segmentation results of all numbers, if a segmentation result ends with a number, cut off the number part, if the non-numeric part is a subset of other segmentation results or the same as other segmentation results, exclude this part of the segmentation results; if a segmentation result contains a space, after removing the space, it is a subset of other segmentation results or equal to other segmentation results, exclude this part of the segmentation results; 5) Calculate the similarity of the key-value word pairs, and retain the i English translations with the highest similarity, i≥5; Step D, extracting Chinese words from the optimized Chinese-English keyword translation word library and the Chinese-English annotation translation word library, and constructing a Chinese word list; The step D specifically includes: extracting the key values ​​of the optimized Chinese-English keyword translation word library and the Chinese-English annotation translation word library to obtain Chinese words, integrating and removing duplicates of the extracted Chinese words, and constructing a Chinese word list; Step E, optimizing the Chinese and English title translation word library; The step E specifically includes: traversing the words in the Chinese vocabulary, searching for translation results in the Chinese and English title translation library, cleaning and summarizing all translation results, and constructing the Chinese and English title translation library for query matching; Step F, translating the Chinese word list through the Chinese-English title translation word library, the Chinese-English keyword translation word library, and the Chinese-English annotation translation word library to construct a Chinese-English bilingual dictionary; The step F specifically includes: traversing the Chinese words in the Chinese vocabulary, searching for corresponding values ​​from the key-value of the three translation libraries: the Chinese-English title translation vocabulary, the Chinese-English keyword translation vocabulary, and the Chinese-English annotation translation vocabulary, and integrating and removing duplicates of the translation results to form synonym pairs of the Chinese dictionary, and entering them into the database to construct a Chinese-English bilingual dictionary.

Images