Corpus generation method, device, system, equipment and readable storage medium

Abstract

The embodiment of the disclosure discloses a corpus generation method, device, system, equipment and readable storage medium, the corpus generation method comprises: obtaining literary works of different languages and feature data of the literary works; matching the literary works of different languages according to the feature data of the literary works to obtain literary work pairs of different languages; and obtaining corpus pairs of different languages by using the literary work pairs of different languages. The characteristics of the literary works of different languages can be utilized, and the literary works can be matched by combining titles, authors, abstracts and chapter names of the literary works to obtain literary work pairs of different languages, so that high-quality corpus pairs can be efficiently and accurately generated from literary works of different languages.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a corpus generation method, apparatus, system, device, and readable storage medium. Background Art

[0002] In recent years, with the increase in international communication, machine translation has played an increasingly important role in fields such as e-commerce, healthcare, and literature. In machine translation, multilingual parallel corpora play a fundamental role in training machine translation models. With the rapid development of the internet and the deepening globalization of trade, obtaining multilingual corpora from the internet has become a crucial source of data for various corpus systems. For example, corpora can come from subtitle files or sentence pairs from foreign language learning websites. However, the quantity of these types of corpora is relatively small. Literary works such as novels and scripts contain vast amounts of data. The rapid development of mobile internet has made it easier for more literary creators to publish their works online. Literary works are a crucial source of corpus data, both in terms of quantity and the continuous updating of data. Summary of the Invention

[0003] In order to solve the problems in the related art, the embodiments of the present disclosure provide a corpus generation method, apparatus, system, device and readable storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for generating corpus, comprising:

[0005] Obtain literary works in different languages ​​and characteristic data of literary works;

[0006] matching literary works in different languages ​​according to the characteristic data of the literary works to obtain pairs of literary works in different languages;

[0007] Corpus pairs in different languages ​​are obtained by using the pairs of literary works in different languages.

[0008] In conjunction with the first aspect, the present disclosure, in a first implementation of the first aspect, further includes:

[0009] The characteristic data of the literary work includes: one or more of the title, author, abstract, and chapter name of the literary work.

[0010] In combination with the first implementation of the first aspect, in the second implementation of the first aspect of the present disclosure, matching literary works in different languages ​​according to the feature data of the literary works to obtain pairs of literary works in different languages ​​includes:

[0011] Obtain the mapping relationship between the titles of literary works in different languages;

[0012] Obtain mapping relationships between authors of literary works in different languages;

[0013] Literary works in different languages ​​are matched according to the mapping relationship between the titles and the mapping relationship between the authors.

[0014] In combination with the second implementation of the first aspect, in a third implementation of the first aspect of the present disclosure, the method further includes:

[0015] Calculate the similarity between titles of literary works in different languages;

[0016] Calculate the matching degree between authors of literary works in different languages;

[0017] The mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, and the matching degree between the authors are weightedly calculated in a preset manner to obtain pairs of literary works in different languages.

[0018] In combination with the third implementation of the first aspect, in a fifth implementation of the first aspect of the present disclosure, the calculating the similarity between the titles of literary works in different languages ​​includes:

[0019] Calculating a first similarity score of similarity between titles of literary works in different languages ​​according to a preset multi-language translation dictionary;

[0020] Translate the titles of literary works in different languages ​​according to a preset translation tool to obtain translation results;

[0021] Perform word segmentation and overlap calculation on the translation results and the titles of the literary works to be matched, and calculate a second similarity score of the similarity between the titles of the literary works in different languages;

[0022] A weighted calculation is performed on the first similarity score and the second similarity score to obtain the similarity between the titles of literary works in different languages.

[0023] In combination with the third implementation of the first aspect, in a fifth implementation of the first aspect of the present disclosure, calculating the matching degree between authors of literary works in different languages ​​includes:

[0024] Transliterating the authors of literary works in different languages ​​according to a preset phonetic scheme to obtain transliterated author names;

[0025] The matching degree between the authors of the literary works in different languages ​​is calculated based on whether the transliterated author name is consistent with the author name of the literary work to be matched.

[0026] In combination with the third implementation of the first aspect, in a sixth implementation of the first aspect of the present disclosure, the method further includes:

[0027] Computing similarities between abstracts of literary works in different languages; and / or

[0028] Calculate the similarity between the content of specific parts of literary works in different languages;

[0029] A weighted calculation is performed on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, the matching degree between the authors, the similarity between the abstracts, and / or the similarity between the contents of specific parts to obtain pairs of literary works in different languages.

[0030] In combination with the first implementation of the first aspect, in a seventh implementation of the first aspect of the present disclosure, obtaining a corpus pair in a different language by using the pair of literary works in different languages ​​includes:

[0031] aligning chapters of literary works in different languages ​​in a pair of literary works according to chapter contents and chapter names of the literary works;

[0032] Generate sentence groups containing corresponding sentences in different languages ​​from aligned passages in literary works;

[0033] According to a preset sentence group screening rule, the sentence groups are screened based on the length ratio and / or similarity between corresponding sentences in the sentence groups;

[0034] Corpus pairs are generated based on the filtered sentence groups that meet the sentence group screening rules.

[0035] In conjunction with the seventh implementation of the first aspect, in an eighth implementation of the first aspect of the present disclosure, aligning the chapters of the literary works in different languages ​​in the literary work pair according to the chapter content and chapter names of the literary works includes:

[0036] Segment the content of different languages ​​in literary works;

[0037] Translate words in the chapter of the translated literary work according to a preset multi-language translation dictionary to count the number of identical n-grams between the chapter of the translated literary work and the chapter of the literary work to be matched;

[0038] Sorting the chapter groups consisting of the chapters of the translated literary work and the chapters of the literary work to be matched from high to low according to the number of identical n-grams, and selecting the top k chapter groups from the sorted chapter groups as high similarity chapter groups, where k is a positive integer;

[0039] Calculating the similarity of chapter titles in different languages ​​in the high-similarity chapter group;

[0040] The chapter group with the highest similarity in chapter name is selected from the high-similarity chapter groups as the aligned chapters.

[0041] In combination with the eighth implementation manner of the first aspect, in a ninth implementation manner of the first aspect of the present disclosure, calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter group includes:

[0042] Normalizing the chapter numbers in the chapter names of different languages ​​in the high-similarity chapter group to calculate the absolute value of the difference between the chapter numbers in different languages; and / or

[0043] Calculating the similarity between the chapter titles in the chapter names of different languages ​​in the high-similarity chapter group;

[0044] The similarity of the chapter names in different languages ​​in the high-similarity chapter group is calculated according to the absolute value of the difference between the chapter numbers in different languages ​​and / or the similarity between the chapter titles in the chapter names in different languages.

[0045] In a second aspect, an embodiment of the present disclosure provides a corpus generation device, comprising:

[0046] an acquisition module configured to acquire literary works in different languages ​​and feature data of the literary works;

[0047] a matching module configured to match literary works in different languages ​​according to the feature data of the literary works to obtain pairs of literary works in different languages;

[0048] The obtaining module is configured to obtain a corpus pair in a different language by using the pair of literary works in different languages.

[0049] In a third aspect, an embodiment of the present disclosure provides a method for generating corpus, including:

[0050] Obtaining first-language literary works, first-language literary work feature data including the title, author, abstract, and chapter names of the first-language literary works, and second-language literary works, second-language literary work feature data including the title, author, abstract, and chapter names of the second-language literary works;

[0051] Obtaining a mapping relationship between the titles of the first language literary works and the titles of the second language literary works, as well as a mapping relationship between the authors;

[0052] Calculating the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts;

[0053] performing a weighted calculation on the mapping relationship between the titles of the first language literary work and the second language literary work, the mapping relationship between the authors, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts to obtain a literary work pair including the first language literary work and the second language literary work;

[0054] Aligning the chapters in the first-language literary work and the second-language literary work according to the chapter content and chapter names in the first-language literary work and the second-language literary work in the literary work pair;

[0055] Obtain sentence pairs containing corresponding first language sentences and second language sentences from the aligned texts;

[0056] The sentence pairs are screened according to preset corpus screening rules to generate corpus pairs.

[0057] In conjunction with the third aspect, in a first implementation of the third aspect, the present disclosure calculates the similarity between the titles of the first language literary work and the second language literary work, including:

[0058] Calculating a first similarity score for the similarity between the title of the literary work in the first language and the title of the literary work in the second language according to a preset mutual translation dictionary between the first language and the second language;

[0059] Translate the title of the first language literary work according to a preset translation tool to obtain a translation result;

[0060] Performing word segmentation and overlap calculation on the translation result and the title of the second language literary work to be matched, and calculating a second similarity score of the similarity between the title of the first language literary work and the title of the second language literary work;

[0061] A weighted calculation is performed on the first similarity score and the second similarity score to obtain the similarity between the title of the first language literary work and the title of the second language literary work.

[0062] In conjunction with the third aspect, in a second implementation of the third aspect of the present disclosure, calculating the matching degree between the authors of the first language literary work and the second language literary work includes:

[0063] Transliterate the author of the first language literary work according to a preset phonetic scheme to obtain the transliterated author name;

[0064] The matching degree between the authors of the first language literary work and the second language literary work is calculated based on whether the transliterated author name is consistent with the author name of the second language literary work to be matched.

[0065] In conjunction with the third aspect, in a third implementation of the third aspect of the present disclosure, screening the sentence pairs according to a preset corpus screening rule to generate corpus pairs includes:

[0066] According to a preset sentence pair screening rule, the sentence pairs are screened based on the length ratio and / or similarity between corresponding sentences in the sentence pairs;

[0067] Corpus pairs are generated based on the filtered sentence pairs that meet the sentence pair screening rules.

[0068] In conjunction with the third aspect, in a fourth implementation of the third aspect of the present disclosure, aligning chapters in the first-language literary work and the second-language literary work in the literary work pair based on chapter content and chapter names includes:

[0069] Segment the text content of the first language literary work and the second language literary work in the literary work pair;

[0070] translating words in a passage of the first language literary work according to a preset mutual translation dictionary between the first language and the second language to count the number of identical n-grams in the passage of the first language literary work and the passage of the second language literary work;

[0071] Sorting the passage pairs consisting of the passages of the first language literary work and the passages of the second language literary work from high to low according to the number of identical n-grams, and selecting the top k passage pairs from the sorted passage pairs as the high-similarity passage pairs, where k is a positive integer;

[0072] Calculating the similarity between the chapter title in the first language and the chapter title in the second language in the high-similarity chapter pair;

[0073] The chapter pairs with the highest similarity in chapter name are selected from the high-similarity chapter pairs as aligned chapters.

[0074] In combination with the fourth implementation of the third aspect, in a fifth implementation of the third aspect of the present disclosure, calculating the similarity between the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair includes:

[0075] Normalizing the chapter numbers in the first language chapter name and the second language chapter name in the high-similarity chapter pair to calculate the absolute value of the difference between the chapter numbers in the first language chapter name and the chapter numbers in the second language chapter name; and / or

[0076] Calculating the similarity between the chapter title in the first language and the chapter title in the second language in the high-similarity chapter pair;

[0077] The similarity between the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair is calculated based on the absolute value of the difference between the chapter number in the chapter name in the first language and the chapter number in the chapter name in the second language and / or the similarity between the chapter title in the chapter name in the first language and the chapter title in the chapter name in the second language.

[0078] In a fourth aspect, an embodiment of the present disclosure provides a corpus generation device, comprising:

[0079] A first acquisition module is configured to acquire a first language literary work, first language literary work feature data including the title, author, abstract, and chapter name of the first language literary work, and a second language literary work, second language literary work feature data including the title, author, abstract, and chapter name of the second language literary work;

[0080] a second acquisition module configured to acquire a mapping relationship between the titles of the first language literary works and the titles of the second language literary works, and a mapping relationship between the authors;

[0081] a first calculation module configured to calculate the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts, and the similarity between specific parts of the contents;

[0082] a second calculation module configured to perform weighted calculation on a mapping relationship between the titles of the first language literary work and the second language literary work, a mapping relationship between the authors, similarity between the titles, matching degrees between the authors, similarity between the abstracts, and similarity between specific portions of content, to obtain a literary work pair comprising the first language literary work and the second language literary work;

[0083] an alignment module configured to align chapters in the first language literary work and the second language literary work in the literary work pair according to chapter contents and chapter names in the first language literary work and the second language literary work;

[0084] a third acquisition module configured to acquire sentence pairs including corresponding sentences in the first language and sentences in the second language from the aligned passages;

[0085] The generating module is configured to screen the sentence pairs according to a preset corpus screening rule to generate corpus pairs.

[0086] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor; wherein,

[0087] The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the first aspect, the first to ninth implementations of the first aspect, the third aspect, and the first to fifth implementations of the third aspect.

[0088] In the sixth aspect, an embodiment of the present disclosure provides a readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in any one of the first aspect, the first to ninth implementations of the first aspect, the third aspect, and the first to fifth implementations of the third aspect is implemented.

[0089] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0090] According to the technical solution provided by the embodiments of the present disclosure, literary works in different languages ​​and feature data of the literary works are obtained; the literary works in different languages ​​are matched according to the feature data of the literary works to obtain pairs of literary works in different languages; and corpus pairs in different languages ​​are obtained using the pairs of literary works in different languages. The characteristics of the literary works in different languages ​​can be used to match the literary works in combination with the titles, authors, abstracts and chapter names of the literary works to obtain pairs of literary works in different languages, thereby automatically generating high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0091] According to the technical solution provided by the embodiments of the present disclosure, the characteristic data of the literary works include one or more of the title, author, abstract, and chapter name of the literary works. The characteristics of literary works in different languages ​​can be utilized to combine the title, author, abstract and chapter name of the literary works to match the literary works, obtain pairs of literary works in different languages, and thus automatically generate high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0092] According to the technical solution provided by the embodiments of the present disclosure, literary works in different languages ​​are matched according to the characteristic data of the literary works to obtain pairs of literary works in different languages, including: obtaining a mapping relationship between the titles of literary works in different languages; obtaining a mapping relationship between the authors of literary works in different languages; matching literary works in different languages ​​according to the mapping relationship between the titles and the mapping relationship between the authors. The characteristics of literary works in different languages ​​can be utilized to match literary works in combination with the titles, authors, abstracts and chapter names of literary works to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0093] According to the technical solution provided by the embodiment of the present disclosure, the method also includes: calculating the similarity between the titles of literary works in different languages; calculating the matching degree between the authors of literary works in different languages; performing weighted calculations on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, and the matching degree between the authors in a preset manner to obtain pairs of literary works in different languages. The characteristics of literary works in different languages ​​can be utilized to match literary works in combination with the titles, authors, abstracts, and chapter names of literary works to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0094] According to the technical solution provided by the embodiment of the present disclosure, the similarity between the titles of literary works in different languages ​​is calculated, including: calculating a first similarity score for the similarity between the titles of literary works in different languages ​​according to a preset mutual translation dictionary of multiple languages; translating the titles of literary works in different languages ​​according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the titles of the literary works to be matched, and calculating a second similarity score for the similarity between the titles of literary works in different languages; performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the titles of literary works in different languages. The characteristics of literary works in different languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0095] According to the technical solution provided by the embodiments of the present disclosure, the matching degree between the authors of literary works in different languages ​​is calculated by: transliterating the authors of literary works in different languages ​​according to a preset phonetic notation scheme to obtain the transliterated author name; calculating the matching degree between the authors of literary works in different languages ​​according to whether the transliterated author name is consistent with the author name of the literary work to be matched. The characteristics of literary works in different languages ​​can be utilized to match the literary works in combination with the title, author, abstract and chapter name of the literary work to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0096] According to the technical solution provided by the embodiment of the present disclosure, the method also includes: calculating the similarity between the abstracts of literary works in different languages; and / or calculating the similarity between the contents of specific parts of literary works in different languages; performing weighted calculations on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, the matching between the authors, the similarity between the abstracts, and / or the similarity between the contents of specific parts to obtain pairs of literary works in different languages. The characteristics of literary works in different languages ​​can be utilized to combine the titles, authors, abstracts and chapter names of literary works to match literary works and obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0097] According to the technical solution provided by the embodiment of the present disclosure, corpus pairs in different languages ​​are obtained by utilizing the pairs of literary works in different languages, including: aligning the chapters of the literary works in different languages ​​in the pair of literary works according to the chapter content and chapter names of the literary works; generating sentence groups containing corresponding sentences in different languages ​​from the aligned chapters in the literary works; screening the sentence groups based on the length ratio and / or similarity between the corresponding sentences in the sentence groups according to preset sentence group screening rules; generating corpus pairs based on the screened sentence groups that meet the sentence group screening rules, and utilizing the characteristics of literary works in different languages, combining the titles, authors, abstracts and chapter names of literary works to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0098] According to the technical solution provided by the embodiment of the present disclosure, chapters of literary works in different languages ​​in a literary work pair are aligned according to the chapter contents and chapter names of the literary works, including: segmenting the chapter contents of different languages ​​in the literary work pair; translating the words in the chapters of the translated literary works according to a preset multi-language translation dictionary to count the number of identical n-grams in the chapters of the translated literary works and the chapters of the literary works to be matched; and aligning the chapter groups composed of the chapters of the translated literary works and the chapters of the literary works to be matched according to the number of identical n-grams. The number of m is sorted from high to low, and the first k chapter groups are selected from the sorted chapter groups as high-similarity chapter groups, where k is a positive integer; the similarity of the chapter names in different languages ​​in the high-similarity chapter groups is calculated; the chapter group with the highest similarity in the chapter name is selected from the high-similarity chapter groups as the aligned chapters, and the characteristics of literary works in different languages ​​can be utilized, combined with the title, author, abstract and chapter name of the literary works to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0099] According to the technical solution provided by the embodiment of the present disclosure, by calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter group, including: normalizing the chapter numbers in the chapter names in different languages ​​in the high-similarity chapter group to calculate the absolute value of the difference between the chapter numbers in different languages; and / or calculating the similarity between the chapter titles in the chapter names in different languages ​​in the high-similarity chapter group; calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter group based on the absolute value of the difference between the chapter numbers in different languages ​​and / or the similarity between the chapter titles in the chapter names in different languages, the characteristics of literary works in different languages ​​can be utilized, and literary works matching and chapter alignment can be performed in combination with the title, author, abstract and chapter name of the literary works, so as to automatically generate high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0100] According to the technical solution provided by the embodiments of the present disclosure, the acquisition module is configured to acquire literary works in different languages ​​and feature data of the literary works; the matching module is configured to match the literary works in different languages ​​based on the feature data of the literary works to obtain pairs of literary works in different languages; the acquisition module is configured to use the pairs of literary works in different languages ​​to obtain corpus pairs in different languages. The characteristics of literary works in different languages ​​can be used to match the literary works in combination with the titles, authors, abstracts and chapter names of the literary works to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0101] According to the technical solution provided by the embodiment of the present disclosure, by obtaining a first language literary work, first language literary work feature data including the title, author, abstract and chapter name of the first language literary work, a second language literary work and second language literary work feature data including the title, author, abstract and chapter name of the second language literary work; obtaining the mapping relationship between the title of the first language literary work and the title of the second language literary work and the mapping relationship between the authors; calculating the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts; and performing the mapping relationship between the titles of the first language literary work and the second language literary work, the mapping relationship between the authors, The similarity between titles, the matching degree between authors, the similarity between abstracts and the similarity between the contents of specific parts are weightedly calculated to obtain a pair of literary works containing literary works in the first language and literary works in the second language; the chapters in the first language literary works and the second language literary works in the literary work pairs are aligned according to the chapter content and chapter names in the first language literary works and the second language literary works; sentence pairs containing corresponding sentences in the first language and sentences in the second language are obtained from the aligned chapters; the sentence pairs are screened according to preset corpus screening rules to generate corpus pairs, and the characteristics of the literary works in the two languages ​​can be used to combine the titles, authors, abstracts and chapter names of the literary works to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0102] According to the technical solution provided by the embodiment of the present disclosure, the similarity between the titles of literary works in a first language and literary works in a second language is calculated, including: calculating a first similarity score of the similarity between the titles of the literary works in the first language and the titles of the literary works in the second language according to a preset mutual translation dictionary between the first language and the second language; translating the title of the literary work in the first language according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the title of the second language literary work to be matched, and calculating a second similarity score of the similarity between the title of the literary work in the first language and the title of the literary work in the second language; performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the title of the literary work in the first language and the title of the literary work in the second language. The characteristics of the literary works in the two languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from the literary works in the two languages.

[0103] According to the technical solution provided by the embodiments of the present disclosure, by calculating the matching degree between the authors of a first-language literary work and a second-language literary work, including: transliterating the author of the first-language literary work according to a preset phonetic notation scheme to obtain a transliterated author name; calculating the matching degree between the authors of the first-language literary work and the second-language literary work based on whether the transliterated author name is consistent with the author name of the second-language literary work to be matched, the characteristics of the literary works in the two languages ​​can be utilized, and the titles, authors, abstracts, and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from the literary works in the two languages.

[0104] According to the technical solution provided by the embodiments of the present disclosure, the sentence pairs are screened according to the preset corpus screening rules to generate corpus pairs, including: screening the sentence pairs based on the length ratio and / or similarity between corresponding sentences in the sentence pairs according to the preset sentence pair screening rules; generating corpus pairs based on the screened sentence pairs that meet the sentence pair screening rules, and utilizing the characteristics of literary works in two languages, combining the titles, authors, abstracts and chapter names of the literary works to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from the literary works in two languages.

[0105] According to the technical solution provided by the embodiment of the present disclosure, the chapters in the first language literary work and the second language literary work in the literary work pair are aligned according to the chapter contents and chapter names in the first language literary work and the second language literary work, including: word segmentation of the chapter contents of the first language literary work and the second language literary work in the literary work pair; translation of words in the chapters of the first language literary work according to a preset mutual translation dictionary between the first language and the second language to count the number of identical n-grams in the chapters of the first language literary work and the chapters of the second language literary work; and comparison of the n-grams of the chapters of the first language literary work and the chapters of the second language literary work. The chapter pairs composed of the chapters are sorted from high to low according to the number of identical n-grams, and the top k chapter pairs are selected from the sorted chapter pairs as high-similarity chapter pairs, where k is a positive integer; the similarity between the chapter names in the first language and the chapter names in the second language in the high-similarity chapter pairs is calculated; the chapter pair with the highest similarity in the chapter name is selected from the high-similarity chapter pairs as the aligned chapters, and the characteristics of the literary works in the two languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0106] According to the technical solution provided by the embodiment of the present disclosure, the similarity between the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair is calculated, including: normalizing the chapter number in the chapter name in the first language and the chapter number in the chapter name in the second language in the high-similarity chapter pair to calculate the absolute value of the difference between the chapter number in the chapter name in the first language and the chapter number in the chapter name in the second language; and / or calculating the difference between the chapter title in the chapter name in the first language and the chapter title in the chapter name in the second language in the high-similarity chapter pair. similarity; according to the absolute value of the difference between the chapter number in the chapter name of the first language and the chapter number in the chapter name of the second language and / or the similarity between the chapter title in the chapter name of the first language and the chapter title in the chapter name of the second language, the similarity between the chapter name of the first language and the chapter title in the chapter name of the second language in the high-similarity chapter pair is calculated, and the characteristics of the literary works in the two languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0107] According to the technical solution provided by the embodiment of the present disclosure, a first acquisition module is configured to acquire first language literary works, first language literary work feature data including the title, author, abstract and chapter name of the first language literary work, and second language literary works, and second language literary work feature data including the title, author, abstract and chapter name of the second language literary work; a second acquisition module is configured to acquire the mapping relationship between the title of the first language literary work and the title of the second language literary work, and the mapping relationship between the authors; a first calculation module is configured to calculate the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts; a second calculation module is configured to calculate the mapping relationship between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts; The method further comprises the following steps: performing weighted calculation on the mapping relationship between the two, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts to obtain a literary work pair containing a first-language literary work and a second-language literary work; an alignment module is configured to align the chapters in the first-language literary work and the second-language literary work in the literary work pair according to the chapter contents and chapter names in the first-language literary work and the second-language literary work; a third acquisition module is configured to obtain sentence pairs containing corresponding sentences in the first language and sentences in the second language from the aligned chapters; a generation module is configured to screen the sentence pairs according to preset corpus screening rules to generate corpus pairs, which can utilize the characteristics of the literary works in the two languages ​​and combine the titles, authors, abstracts and chapter names of the literary works to perform literary work matching and chapter alignment, thereby automatically generating high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0108] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the drawings:

[0110] Figure 1 A flowchart showing a corpus generation method according to an embodiment of the present disclosure is shown;

[0111] Figure 2 A structural block diagram of a corpus generation device according to an embodiment of the present disclosure is shown;

[0112] Figure 3 A schematic diagram showing an exemplary application of generating a corpus pair using a bilingual version of a novel using the corpus generation method according to an embodiment of the present disclosure;

[0113] Figure 4 A flowchart showing a corpus generation method according to another embodiment of the present disclosure is shown;

[0114] Figure 5 A structural block diagram of a corpus generation device according to another embodiment of the present disclosure is shown;

[0115] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0116] Figure 7 It is a structural diagram of a computer system suitable for implementing the corpus generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0117] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0118] In the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the presence of labels, numbers, steps, actions, components, parts, or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other labels, numbers, steps, actions, components, parts, or combinations thereof exist or are added.

[0119] It should also be noted that, in the absence of conflict, the embodiments and labels in the embodiments of the present disclosure can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0120] According to the technical solution provided by the embodiments of the present disclosure, literary works in different languages ​​and feature data of the literary works are obtained; the literary works in different languages ​​are matched according to the feature data of the literary works to obtain pairs of literary works in different languages; and corpus pairs in different languages ​​are obtained using the pairs of literary works in different languages. The characteristics of the literary works in different languages ​​can be used to match the literary works in combination with the titles, authors, abstracts and chapter names of the literary works to obtain pairs of literary works in different languages, thereby automatically generating high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0121] Figure 1 FIG. 1 is a flow chart of a corpus generation method according to an embodiment of the present disclosure. Figure 1 As shown, the corpus generation method includes the following steps S110, S120 and S130:

[0122] In step S110 , literary works in different languages ​​and feature data of the literary works are obtained.

[0123] In step S120, literary works in different languages ​​are matched according to the feature data of the literary works to obtain pairs of literary works in different languages.

[0124] In step S130, corpus pairs in different languages ​​are obtained using the pairs of literary works in different languages.

[0125] In one embodiment of the present disclosure, a literary work may be a sentence or a combination of multiple sentences. A literary work may be composed of sentences, paragraphs, and / or chapters. In one embodiment of the present disclosure, a literary work may refer to various forms of literary texts composed of words, such as novels, poems, and plays.

[0126] In one embodiment of the present disclosure, "different languages" refers to the existence of multiple language versions. Alternatively, "different languages" can refer to multiple natural languages, and literary works in different languages ​​can refer to texts expressed in multiple natural languages. For example, a novel written in Chinese is translated into English and Japanese versions, resulting in three language versions. In one embodiment of the present disclosure, these three language versions can be referred to as parallel texts or matching literary works, i.e., literary works with consistent content but expressed in different languages. Broadly speaking, matching literary works also include translated materials with similar content to the original text, primarily used to provide a deeper understanding of the original text. From this perspective, matching literary works can include any reference material with content close to the original text. In one embodiment of the present disclosure, two matching literary works in different languages ​​can constitute a literary work pair. For example, the Chinese, English, and Japanese versions of the novel described above can constitute three literary work pairs: a Chinese-English literary work pair, a Chinese-Japanese literary work pair, and an English-Japanese literary work pair.

[0127] In one embodiment of the present disclosure, the characteristic data of a literary work may refer to data that characterizes the characteristics of the literary work so that the current literary work can be distinguished from other literary works. In one embodiment of the present disclosure, the characteristic data of the literary work includes: one or more of the title, author, abstract, and chapter name of the literary work. For example, the title of a Chinese literary work is "A Pen", and the title of the parallel (or corresponding) English literary work is "APen". Therefore, the characteristic data of the Chinese literary work includes the title "A Pen", and the characteristic data of the English literary work includes the title "A Pen".

[0128] According to the technical solution provided by the embodiments of the present disclosure, the characteristic data of the literary works include one or more of the title, author, abstract, and chapter name of the literary works. The characteristics of literary works in different languages ​​can be utilized to combine the title, author, abstract and chapter name of the literary works to match the literary works, obtain pairs of literary works in different languages, and thus automatically generate high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0129] In one embodiment of the present disclosure, the characteristic data of a literary work is not limited to the title, author, abstract, and chapter name of the literary work, but may also include characteristics that characterize the literary work, such as paragraph structure and number. This disclosure will not elaborate on this.

[0130] In one embodiment of the present disclosure, corresponding feature data may be obtained for each language version of the obtained literary works. For example, when there are Chinese literary works, English literary works, and Japanese literary works, the feature data of the Chinese literary works and the English literary works may be obtained, the feature data of the English literary works and the English literary works may be obtained, and the feature data of the Japanese literary works and the Japanese literary works may be obtained.

[0131] In one embodiment of the present disclosure, in order to generate corpora in different languages, it is necessary to match (align) literary works in multiple language versions. For example, at least one of the titles, authors, abstracts, and chapter names of the literary works in the database can be used for alignment to obtain pairs of literary works in different languages. In one embodiment of the present disclosure, the titles can be used to match literary works in different languages. For example, the title of the Chinese literary work is "A Pen" and the title of the English literary work is "A Pen". It can be determined through Chinese-English translation that "A Pen" is consistent with "A Pen". The Chinese literary work with the title "A Pen" is thus matched with the English literary work with the title "A Pen" to generate a pair of literary works containing the corresponding Chinese literary work and English literary work.

[0132] It should be noted that matching literary works using only one of the following: title, author, abstract, or chapter name may result in matching failures. For example, if a Chinese literary work database stores multiple literary works titled "A Pen" written by different authors, and an English literary work database stores a literary work titled "A Pen" translated from multiple literary works titled "A Pen," the content of these literary works differs from each other, making matching difficult. Consequently, it is impossible to obtain a literary work pair containing the corresponding Chinese and English literary works. The same problem may also exist when matching literary works using only one of the following: author, abstract, or chapter name.

[0133] Therefore, multiple items from the titles, authors, abstracts, and chapter names of the literary works in the literary work database can be used to match literary works, thereby achieving more accurate literary work matching. For example, if a Chinese literary work database stores multiple literary works titled "A Pen" written by different authors, and the author of one of the literary works is "Zhang San"; and at the same time, an English literary work database stores literary works titled "A Pen" translated from multiple literary works titled "A Pen," and the author of one of the literary works is the English text "ZHANG San" obtained by transliterating the Chinese characters "ZHANG San" through Chinese pinyin, then the literary work titled "A Pen" and authored by "ZHANG San" can be matched with the literary work titled "A Pen" and authored by "ZHANG San," thereby obtaining a matching literary work pair containing both Chinese and English versions. Therefore, using multiple items from the titles, authors, abstracts, and chapter names of the literary works in the literary work database to match literary works in multiple languages ​​can improve the accuracy of literary work matching. In addition, those skilled in the art will appreciate that the author name of a literary work can be transliterated or the author's corresponding name in various languages. For example, based on the above example, if the author of a Chinese literary work is "张三", and the English author name used when the literary work of author "张三" is translated into English is "Tim ZHANG", then the literary work titled "A Pen" and authored by "张三" can be matched with the literary work titled "A Pen" and authored by "Tim ZHANG", thereby obtaining a literary work pair containing both Chinese and English versions of the literary work.

[0134] In one embodiment of the present disclosure, the corpus pairs to be generated can be corpus pairs in different languages, and the corpus pairs correspond to sentence groups. For example, a corpus pair (bilingual corpus) refers to the same sentence described in two different languages. This pair of sentences (sentence pair or sentence group) is the bilingual corpus. For example, "I love China" and "I love China" are a bilingual corpus. Therefore, after completing the literary work matching to obtain the literary work pair, it is necessary to align the content of the literary work pair to obtain the aligned sentence group to generate the corpus pair.

[0135] In one embodiment of the present disclosure, different symbols may exist in the same language. For example, the simplified Chinese "I love China" and the traditional Chinese "I love China" are different symbolic expressions of the same textual meaning in the same language. In some cases, literary works in simplified Chinese and traditional Chinese use the same grammatical expression. For example, novels written by modern writers can be considered to be the same as literary works in simplified Chinese without the need to generate corpus pairs. In other cases, literary works in simplified Chinese and traditional Chinese do not use the same grammatical expression. For example, literary works in traditional Chinese written by ancient writers and texts that have been translated into modern characters are difficult to directly determine to be the same due to the inconsistency in the grammatical expression between the two. In this case, it may be necessary to generate corpus pairs. For example, due to differences in wording and grammatical expressions, the traditional Chinese poem "蒹葭蒼蒼,白露為霜" and its simplified modern translation "Luwei dense and green, crystal dew condensed into frost" may need to be matched based on feature data such as the poem's title, author, and chapter name to output a corpus pair. For another example, for the traditional Chinese poem "蒹葭蒼蒼,白露為霜" and its English translation "The young shoots of reeds are grey, The white dew looks like frost," a weighted calculation may be performed on the mapping relationship between poem titles, the similarity between titles, the mapping relationship between authors, the matching degree between authors, the similarity between abstracts, and / or the similarity between specific parts of content to obtain pairs of literary works in different languages. This is how bilingual sentence pairs (sentence pairs, corpus pairs) representing this poem can be obtained. Similarly, we can obtain a bilingual sentence pair (sentence pair, corpus pair) of the simplified Chinese poem "The young shoots of reeds are grey,The whitedew looks like frost" and its English translation "The young shoots of reeds are grey,The whitedew looks like frost".

[0136] It can be seen from the above embodiments that for a variety of symbol systems that may exist in a language, for example, Chinese includes symbol systems such as simplified Chinese, traditional Chinese, and Cantonese Chinese, there may be a situation where the expression methods of literary works completed using these symbol systems are too different, so it may be necessary to generate literary work pairs by aligning literary works, and then obtain corpus pairs. In this case, in one embodiment of the present disclosure, in order to obtain corpus pairs, different symbol systems of the same language can be considered as different languages. By analogy, for various languages ​​and the multiple symbol systems of each language itself, the corpus generation method in the embodiment of the present disclosure can be used to generate corpus pairs. Therefore, regardless of the size of the gap between various languages ​​or language symbol systems, the corpus generation method according to the embodiment of the present disclosure can be used to generate corpus pairs. In other words, as long as the feature data of literary works in different languages ​​or different symbol systems of the same language can be extracted to match literary works in different languages, pairs of literary works in different languages ​​can be obtained, and then corpus pairs in different languages ​​can be obtained.

[0137] In one embodiment of the present disclosure, the corpus generation method is illustrated above using literary works in two languages ​​as an example. Those skilled in the art will appreciate that the solution of the embodiment of the present disclosure can be applied to generating corpora in three or more languages.

[0138] In one embodiment of the present disclosure, step S120 includes: obtaining a mapping relationship between titles of literary works in different languages; obtaining a mapping relationship between authors of literary works in different languages; and matching literary works in different languages ​​based on the mapping relationship between titles and the mapping relationship between authors.

[0139] According to the technical solution provided by the embodiments of the present disclosure, literary works in different languages ​​are matched according to the characteristic data of the literary works to obtain pairs of literary works in different languages, including: obtaining a mapping relationship between the titles of literary works in different languages; obtaining a mapping relationship between the authors of literary works in different languages; matching literary works in different languages ​​according to the mapping relationship between titles and the mapping relationship between authors. The characteristics of literary works in different languages ​​can be utilized to match literary works in combination with titles, authors, abstracts and chapter names of literary works to obtain pairs of literary works in different languages, thereby automatically generating high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0140] In one embodiment of the present disclosure, obtaining a mapping relationship between the titles of literary works in different languages ​​may refer to obtaining a mapping relationship between the titles of all or part of the literary works in multiple language versions in a database of literary works in corresponding language versions by accessing the Internet or a specific database, etc. (for example, the mapping relationship between the Chinese name of the novel "Journey to the West" and the English title of the novel "Journey to the West").

[0141] In one embodiment of the present disclosure, obtaining a mapping relationship between authors of literary works in different languages ​​may refer to obtaining the author mapping relationship of multiple language versions of all or part of the literary works in a literary works database of corresponding language versions by accessing the Internet or a specific database, etc. (for example, the mapping relationship between the author "Wu Chengen" of the novel "Journey to the West" and the English author "WU Chengen").

[0142] In one embodiment of the present disclosure, literary works in different languages ​​are matched based on the mapping relationship between titles and the mapping relationship between authors. For example, through the mapping relationship between the titles of literary works "Journey to the West" and "Journey to the West" and the mapping relationship between the authors of the literary work "Journey to the West" and "WU Chengen", pairs of literary works in different languages ​​can be obtained from the Chinese literary work database. In one embodiment of the present disclosure, the mapping relationship of titles can be saved in "title alignment data". In one embodiment of the present disclosure, the mapping relationship of authors can be saved in "author alignment data". In one embodiment of the present disclosure, the "title alignment data" and "author alignment data" can be saved in "literary work matching data". Therefore, the literary work matching data stores the "title alignment data" and "author alignment data" of the literary works in multiple language versions in the determined literary work pairs.

[0143] In one embodiment of the present disclosure, whether the mapping relationship of the titles of literary works in different languages ​​can be obtained from the "title alignment data" refers to whether the titles of literary works in different languages ​​match. If the mapping relationship of the titles of literary works in different languages ​​can be obtained from the "title alignment data", it means that the titles of literary works in different languages ​​match. If the mapping relationship of the titles of literary works in different languages ​​cannot be obtained from the "title alignment data", it means that the titles of literary works in different languages ​​do not match. In this case, obtaining a pair of literary works in different languages ​​according to the mapping relationship of the titles may refer to calculating the literary work matching score using the matching degree score of the titles of literary works in different languages ​​(for example, recorded as title_match_score). For example, if the titles of literary works in different languages ​​match, the title matching score of the literary works in different languages, title_match_score, is 1. For example, if the titles of literary works in different languages ​​do not match, the title matching score of the literary works in different languages, title_match_score, is 0.

[0144] In one embodiment of the present disclosure, whether the mapping relationship of the authors of literary works in different languages ​​can be obtained from the "author alignment data" refers to whether the authors of the literary works in different languages ​​match. If the mapping relationship of the authors of literary works in different languages ​​can be obtained from the "author alignment data", it means that the authors of the literary works in different languages ​​match. If the mapping relationship of the authors of literary works in different languages ​​cannot be obtained from the "author alignment data", it means that the authors of the literary works in different languages ​​do not match. In this case, obtaining a pair of literary works containing different languages ​​according to the mapping relationship of the authors may refer to calculating the literary work matching score using the matching degree score of the authors of literary works in different languages ​​(for example, recorded as author_match_score). For example, if the authors of literary works in different languages ​​match, the author matching score author_match_score of the literary works in different languages ​​is 1. For example, if the authors of literary works in different languages ​​do not match, the author matching score author_match_score of the literary works in different languages ​​is 0.

[0145] For example, when the title matching score title_match_score of the literary works in different languages ​​is 1 and the author matching score author_match_score of the literary works in different languages ​​is 1, it can be determined that the literary works in different languages ​​are matching literary works, and thus a literary work pair of matching literary works can be obtained.

[0146] It should be understood that the above representation and scoring methods for the title matching scores and author matching scores of literary works in different languages ​​are merely examples, and their purpose is to achieve matching of literary works in different languages ​​based on the mapping relationships between the titles and the mapping relationships between the authors. Therefore, those skilled in the art can adopt various methods based on the teachings of this disclosure to obtain pairs of literary works in different languages.

[0147] Those skilled in the art will appreciate that by obtaining the existing mapping relationships of titles and authors, the process of matching literary works can be simplified, and pairs of literary works in different languages ​​can be accurately and quickly obtained.

[0148] In one embodiment of the present disclosure, step S120 also includes: calculating the similarity between the titles of literary works in different languages; calculating the matching degree between the authors of literary works in different languages; and performing weighted calculation on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, and the matching degree between the authors in a preset manner to obtain pairs of literary works in different languages.

[0149] According to the technical solution provided by the embodiment of the present disclosure, literary works in different languages ​​are matched according to the characteristic data of the literary works to obtain pairs of literary works in different languages, and the solution also includes: calculating the similarity between the titles of literary works in different languages; calculating the matching degree between the authors of literary works in different languages; performing weighted calculations on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, and the matching degree between the authors in a preset manner to obtain pairs of literary works in different languages. The characteristics of literary works in different languages ​​can be utilized to match literary works in combination with the titles, authors, abstracts, and chapter names of literary works to obtain pairs of literary works in different languages, thereby automatically generating high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0150] In some cases, it may not be possible to obtain the mapping relationship between the titles of literary works and the mapping relationship between the authors of literary works in multiple language versions of literary works databases. In other cases, due to reliability issues of Internet resources or specific databases, simply obtaining the mapping relationship between existing titles and authors may not be sufficient to accurately match literary works. Therefore, literary work matching can be performed by calculating the similarity between the titles of literary works in different languages ​​and calculating the matching degree between the authors of literary works in different languages.

[0151] In one embodiment of the present disclosure, the similarity between titles of literary works in different languages ​​is calculated, including: calculating a first similarity score for the similarity between titles of literary works in different languages ​​according to a preset translation dictionary of multiple languages; translating the titles of literary works in different languages ​​according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the titles of literary works to be matched, and calculating a second similarity score for the similarity between titles of literary works in different languages; and performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between titles of literary works in different languages.

[0152] According to the technical solution provided by the embodiment of the present disclosure, by calculating the similarity between the titles of literary works in different languages, the method includes: calculating a first similarity score of the similarity between the titles of literary works in different languages ​​according to a preset mutual translation dictionary of multiple languages; translating the titles of literary works in different languages ​​according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the titles of the literary works to be matched, and calculating a second similarity score of the similarity between the titles of literary works in different languages; performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the titles of literary works in different languages, and utilizing the characteristics of literary works in different languages, combining the title, author, abstract and chapter name of the literary works to match the literary works, and obtaining pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0153] In one embodiment of the present disclosure, if the titles of literary works in different languages ​​do not appear in the "title alignment data", it is necessary to calculate and score the similarities of the titles of literary works in different languages. For example, the first similarity score of the similarity between the titles of literary works in different languages ​​can be calculated based on a preset mutual translation dictionary of multiple languages. A mutual translation dictionary may refer to a dictionary that provides a way of expressing various languages ​​to each other, and may include a bilingual dictionary, a trilingual dictionary, or a dictionary of more languages. For example, for a Chinese-English bilingual dictionary, the content included is in the form of a key-value pair: "中国": "China". In one embodiment of the present disclosure, the preset mutual translation dictionary of multiple languages ​​may also include a mutual translation probability dictionary, which refers to a dictionary that expresses words in one language in other languages ​​and gives the probability of mutual translation between word pairs. The following uses the Chinese-English mutual translation probability dictionary as an example to illustrate the method of calculating the first similarity score of the similarity between the titles of Chinese and English literary works. First, the Chinese and English titles are segmented separately, and then the similarity score between the two titles is calculated as the first similarity score, for example, represented by title_similarity_score_1, through a preset Chinese-English translation probability dictionary (which includes the probability of mutual translation between two words in Chinese and English. The probability data can be derived from word alignment statistics of parallel corpus).

[0154] In one embodiment of the present disclosure, when calculating and scoring the similarity of the titles of literary works in different languages, the titles of the literary works in different languages ​​can also be translated according to a preset translation tool to obtain a translation result; the translation result and the title of the literary work to be matched are segmented and the overlap is calculated, and a second similarity score of the similarity between the titles of the literary works in different languages ​​is calculated. The preset translation tool can be used to obtain the translation result, and the second similarity score of the similarity between the titles of the literary works in different languages ​​is calculated. The preset translation tool refers to a machine translation tool, which can be a hardware translation tool, a software translation tool, or a translation tool that combines hardware and software. Taking the preset Chinese-English translation tool as an example, the title A of the Chinese literary work can be translated into English by a translation tool (for example, a machine translation engine), which is recorded as A_1. The translation tool result A_1 and the title B of the English literary work are first subjected to English word segmentation operations, and then the second similarity score of the similarity is calculated by calculating the edit distance of the word segmentation results of A_1 and B and the n-gram (or n-gram model, n-tuple, n-gram syntax, etc.) overlap, for example, represented by title_similarity_score_2. In one embodiment of the present disclosure, n connected words together are n-grams, where n is a positive integer, for example, 1, 2, etc. For example, 2-gram refers to two words that make up the word segmentation result. For example, "I love China" contains 2 2-grams, namely: "I love" and "Love China". Those skilled in the art will understand that the method of calculating the edit distance and n-gram overlap of the word segmentation results of A_1 and B can be carried out in a manner known in the relevant art, and this disclosure will not go into details.

[0155] In one embodiment of the present disclosure, the similarity between titles of literary works in different languages can be calculated by weighting the first similarity score and the second similarity score. Referring to the above example of calculating the similarity between titles of Chinese and English literary works, the first similarity score title_similarity_score_1 and the second similarity score title_similarity_score_2 can be weighted to obtain the similarity between titles of Chinese and English literary works title_similarity_score. For example, the first similarity score title_similarity_score_1 is weighted by w1, and the second similarity score title_similarity_score_2 is weighted by w2, then title_similarity_score = title_similarity_score_1 * w1 + title_similarity_score_2 * w2. In another example, title_similarity_score = (title_similarity_score_1 * w1 + title_similarity_score_2 * w2) / 2. It should be understood that the manner of calculating the similarity between titles of Chinese and English literary works is not limited to the above two examples, and those skilled in the art can employ various manners to weight the first similarity score and the second similarity score to obtain the similarity between titles of literary works in different languages according to the teachings of the embodiments of the present disclosure.

[0156] The manner of calculating the similarity between titles of Chinese and English literary works is discussed above with reference to the example of matching Chinese and English literary works. Those skilled in the art can understand that the similarity between titles of literary works in two languages, three languages or more languages can be calculated in the above manner, and the specific calculation manner can be obtained from the above examples and related technologies, which will not be repeated here.

[0157] In one embodiment of the present disclosure, by the above-mentioned manner of calculating the similarity between titles of literary works in different languages, even if the mapping relationship between the titles of literary works cannot be obtained for the literary works in the literary work database in multiple language versions, or due to the reliability problem of Internet resources or specific databases, only obtaining the mapping relationship between the titles and the mapping relationship between the authors can be insufficient to accurately align the literary works, the similarity between titles of literary works in different languages can be respectively scored and calculated by using the mutual translation dictionary of multiple languages and the preset translation tool, and then the scoring and calculation results are weighted to obtain the similarity between literary work titles.

[0158] In one embodiment of the present disclosure, the degree of matching between authors of literary works in different languages ​​is calculated, including: transliterating the authors of the literary works in different languages ​​according to a preset phonetic notation scheme to obtain a transliterated author name; and calculating the degree of matching between the authors of the literary works in different languages ​​according to whether the transliterated author name is consistent with the author name of the literary work to be matched.

[0159] According to the technical solution provided by the embodiments of the present disclosure, by calculating the matching degree between the authors of literary works in different languages, including: transliterating the authors of literary works in different languages ​​according to a preset phonetic notation scheme to obtain the transliterated author name; calculating the matching degree between the authors of literary works in different languages ​​according to whether the transliterated author name is consistent with the author name of the literary work to be matched, the characteristics of literary works in different languages ​​can be utilized, and the literary works can be matched in combination with the title, author, abstract and chapter name of the literary work to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0160] In one embodiment of the present disclosure, it may be impossible to obtain the mapping relationship of the authors of literary works for literary works in multiple language versions of the literary work database. In another embodiment of the present disclosure, due to reliability issues of Internet resources or specific databases, it may be insufficient to accurately align literary works by simply obtaining the mapping relationship of existing titles and authors. Therefore, it is necessary to translate and match the authors of literary works in different languages ​​to align the authors. In one embodiment of the present disclosure, the titles of literary works in different languages ​​can usually be obtained through translation, while the author names of literary works are usually difficult to translate directly. Therefore, the authors of literary works in different languages ​​can be transliterated according to the preset phonetic notation schemes between multiple languages ​​to obtain the transliterated author names. Taking the translation of the author names of Chinese and English literary works as an example, the English names of some authors can be directly taken from the Chinese pinyin of their Chinese names (the Chinese national standard pinyin scheme can be used). Therefore, if the author pair (author group) of Chinese literary work A and English literary work B does not appear in the "author alignment data" or if there is doubt about the reliability of the author pair in the "author alignment data", the pinyin of the Chinese author of the literary work can be obtained. If the pinyin of the author's name obtained from Chinese literary work A is consistent with the author's name of English literary work B, a matching score for the author's name converted from pinyin to the English name can be obtained, for example, denoted as author_pinyin_match_score. In one embodiment of the present disclosure, if the second-language phonetic notation of the author's name obtained from first-language literary work A is consistent with the author of second-language literary work B, a matching score for the author's name converted from second-language phonetic notation to the second-language name can be obtained, for example, denoted as author_zhuyin_match_score.

[0161] In one embodiment of the present disclosure, the Chinese Pinyin standard can be used as the preset phonetic notation scheme when matching Chinese and English authors of Chinese literary works to English literary works, and other phonetic notation schemes can be used as the preset phonetic notation scheme when matching authors of other languages. When the literary works to be aligned include literary works in three or more different languages, multiple phonetic notation schemes can be used as the preset phonetic notation schemes.

[0162] In one embodiment of the present disclosure, calculating the matching degree between the authors of literary works in different languages ​​based on whether the transliterated author name is consistent with the author name of the literary work to be matched can mean that when the transliterated author name is consistent with the author name to be matched, a matching degree score of 1 is obtained between the author names of the literary works in different languages, and when the transliterated author name is inconsistent with the author name to be matched, a matching degree score of 0 is obtained between the author names of the literary works in different languages. Taking the matching of Chinese and English literary works as an example, if the author name of the Chinese literary work is "张三", the author name is transliterated according to a preset phonetic scheme to obtain the transliterated author name "ZHANG San". If the author name of the English literary work is also "ZHANG San", the matching score author_pinyin_match_score of the author name converted from pinyin to the English name is 1. In another example, if the transliterated author name of the Chinese literary work is "ZHANGSan" and the author name of the English literary work is "LI Si", the matching score author_pinyin_match_score of the author name converted from pinyin to the English name is 0. It should be noted that the matching score of the author's name converted from pinyin and matched with the English name is only one way to calculate the matching degree between authors of literary works in different languages. The matching score between authors of literary works in different languages ​​can also be determined by other methods.

[0163] In one embodiment of the present disclosure, due to the existence of dialects in Chinese or the phonetic notation system different from the national standard Hanyu Pinyin, the way of matching the author names of literary works in different languages can need to consider more complex situations. For example, according to the national standard Hanyu Pinyin notation scheme, the author name "Zhang San" is transliterated to obtain the transliterated author name "ZHANG San", while the author name of the English literary work is "ZOENG Saam", that is, the author name "ZOENG Saam" of the English literary work is the Cantonese notation of "Zhang San". For another example, the author name of the English literary work is "DIONG Sa", that is, the author name "DIONG Sa" of the English literary work is the notation of "Zhang San" in Minnan dialect. Therefore, if the author name is transliterated only according to the national standard Hanyu Pinyin notation scheme, it will result in that the matching score author_pinyin_match_score of the actual matching Chinese author name transliterated into Cantonese Pinyin or Minnan Pinyin and the English name is 0. In order to solve this problem, when performing Chinese-English author matching of Chinese literary works to English literary works, a Hanyu notation scheme including multiple known Hanyu Pinyin schemes can be used, for example, a Pinyin scheme including multiple Hanyu dialects. That is, the matching degree between the authors of literary works in different languages can be calculated according to whether the Chinese transliterated author name obtained by using the Pinyin scheme of multiple Hanyu dialects is consistent with the author name of the literary work to be matched. Similarly, for other languages, if there are different notation rules for the same script, a notation scheme including multiple known notation rules can also be used to match the author name notation and the translated name.

[0164] By analogy, the above embodiment discussed according to the preset notation scheme including multiple known notation rules can be used for matching the similarity between the titles of literary works in different languages, the similarity between the abstracts, and / or the similarity between the contents of specific parts.

[0165] In an embodiment of the present disclosure, in any case where the same script has multiple notation schemes, the notation expression of the script can be obtained according to various notation schemes, and it is determined whether the author names to be matched are consistent. In the case where the notation result needs to be matched, the notation made by various notation schemes for the language can be used for comparison with the language to be matched. It can be understood that in an embodiment of the present disclosure, the notation scheme is a tool for matching literary works in different languages, and after the matching of literary works is completed, the obtained pairs of literary works in different languages can not be limited to the notation scheme.

[0166] In one embodiment of the present disclosure, when a mapping relationship of titles exists in the "title alignment data", the weight of the calculated similarity between titles, title_similarity_score, may be lower than the weight of the mapping relationship of titles, that is, the weight of the title matching score, title_match_score, of the literary work. In one embodiment of the present disclosure, when a mapping relationship of titles does not exist in the "title alignment data", the weight of the calculated similarity between titles, title_similarity_score, may be higher than the weight of the mapping relationship of titles, that is, the weight of the title matching score, title_match_score, of the literary work.

[0167] In one embodiment of the present disclosure, when an author mapping relationship exists in the "author alignment data," the weight of the calculated match score (author_zhuyin_match_score) for matching the author's phonetic notation in the second language with the second language name may be lower than the weight of the author mapping relationship, that is, the weight of the author match score (author_match_score) for the literary work. In one embodiment of the present disclosure, when an author mapping relationship does not exist in the "author alignment data," the weight of the calculated match score (author_zhuyin_match_score) for matching the author's name in the second language with the second language name may be higher than the weight of the author mapping relationship, that is, the weight of the author match score (author_match_score) for the literary work.

[0168] In one embodiment of the present disclosure, a preset method for weighted calculation of the mapping relationship of titles, the calculated similarity between titles, the mapping relationship of authors, and the calculated matching degree between authors may be to use the value obtained by weighted addition of each value as the matching literary work (literary work pair) score of literary works in different languages. In one embodiment of the present disclosure, a preset method for weighted calculation of the mapping relationship of titles, the calculated similarity between titles, the mapping relationship of authors, and the calculated matching degree between authors may be to use the value obtained by weighted addition of each value and then dividing it by a specific value as the matching literary work score of literary works in different languages. That is, according to the teachings of the embodiments of the present disclosure, those skilled in the art may use various methods to calculate the literary works in different languages ​​as matching literary work scores.

[0169] In one embodiment of the present disclosure, a matched literary work score threshold can be set for the matched literary work scores of literary works in different languages. In one embodiment of the present disclosure, when the matched literary work score exceeds the preset matched literary work score threshold, it is determined that the literary works in different languages are matched literary works. In one embodiment of the present disclosure, when the matched literary work score does not exceed the preset matched literary work score threshold, it is determined that the literary works in different languages are not parallel literary works.

[0170] In one embodiment of the present disclosure, step S120 further comprises: calculating the similarity between the summaries of literary works in different languages; and / or calculating the similarity between the contents of specific parts of literary works in different languages; and performing weighted calculation on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, the matching degree between the authors, the similarity between the summaries, and / or the similarity between the contents of specific parts, to obtain a pair of literary works in different languages.

[0171] According to the technical scheme provided by the embodiments of the present disclosure, the literary works in different languages are matched according to the feature data of the literary works to obtain a pair of literary works in different languages, which further comprises: calculating the similarity between the summaries of literary works in different languages; and / or calculating the similarity between the contents of specific parts of literary works in different languages; and performing weighted calculation on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, the matching degree between the authors, the similarity between the summaries, and / or the similarity between the contents of specific parts, to obtain a pair of literary works in different languages. The characteristics of literary works in different languages can be utilized to match literary works by combining titles, authors, summaries and chapter names of literary works, so as to obtain a pair of literary works in different languages, thereby efficiently and accurately generating a pair of high-quality corpora from literary works in different languages.

[0172] In one embodiment of the present disclosure, the literary work features of literary works can further include summaries (or detailed descriptions of literary works). Since the summaries of different literary works in the same language are usually different, the summaries can usually be used to distinguish different literary works in the same language. Therefore, calculating the similarity between the summaries of literary works in different languages can be used to align literary works in different languages, thereby obtaining a pair of literary works containing matched literary works.

[0173] In one embodiment of the present disclosure, the similarity between the abstracts of literary works in different languages ​​can be calculated, for example, represented by abstract_similarity_score. The method of calculating the similarity abstract_similarity_score between the abstracts of literary works is similar to the aforementioned method of calculating the similarity title_similarity_score between titles. In one embodiment of the present disclosure, a third similarity score of the similarity between the abstracts of literary works in different languages ​​can be calculated based on a preset translation dictionary of multiple languages. The following describes a method of calculating the third similarity score of the similarity between the abstracts of Chinese and English literary works using a Chinese-English translation probability dictionary as an example. First, the Chinese and English abstracts are segmented separately, and then the similarity score between the two abstracts is calculated as the third similarity score through a preset Chinese-English translation probability dictionary (which includes the probability of mutual translation between two words in Chinese and English, and the probability data can be derived from word alignment statistics of parallel corpus). For example, it is represented by abstract_similarity_score_3.

[0174] In one embodiment of the present disclosure, when calculating and scoring the similarity of abstracts of literary works in different languages, the translation results of the abstracts of the translated literary works in multiple language versions and the abstracts of the literary works to be matched according to the preset translation tool can also be segmented and the overlap calculation is performed to calculate the fourth similarity score of the similarity between the abstracts of the literary works in different languages. The fourth similarity score can be calculated for the translation results of the abstracts of the literary works in different languages ​​using the preset translation tool. The preset translation tool refers to a machine translation tool, which can be a hardware translation tool, a software translation tool, or a translation tool that combines hardware and software. Taking the preset Chinese-English translation tool as an example, the abstract C of the Chinese literary work can be translated into English by the translation tool (for example, a machine translation engine), which is recorded as C_1. The English word segmentation operation is first performed on the translation tool result C_1 and the abstract D of the English literary work, and then the fourth similarity score of the similarity is calculated by calculating the edit distance and n-gram overlap of the word segmentation results of C_1 and D, for example, represented by abstract_similarity_score_4.

[0175] In one embodiment of the present disclosure, the similarity between abstracts of literary works in different languages ​​can be obtained by weighting the third similarity score and the fourth similarity score. Referring to the above example of calculating the similarity between the abstracts of Chinese and English literary works, the third similarity score abstract_similarity_score_3 and the fourth similarity score abstract_similarity_score_4 can be weighted to obtain the similarity between the abstracts of Chinese and English literary works, abstract_similarity_score. For example, if a weight w3 is set for the third similarity score abstract_similarity_score_3 and a weight w4 is set for the fourth similarity score abstract_similarity_score_4, then abstract_similarity_score = abstract_similarity_score_3*w3 + abstract_similarity_score_4*w4. In another example, abstract_similarity_score = (abstract_similarity_score_3*w3+abstract_similarity_score_4*w4) / 2. It should be understood that the method for calculating the similarity between the abstracts of Chinese and English literary works is not limited to the above two examples. Those skilled in the art can use various methods to perform weighted calculations on the first similarity score and the fourth similarity score based on the teachings of the embodiments of the present disclosure to obtain the similarity between the abstracts of literary works in different languages.

[0176] In one embodiment of the present disclosure, the literary work features of a literary work may further include the content of a specific portion of the literary work (e.g., the first chapter, the last chapter, the first paragraph, or the last paragraph of the literary work). Since the content of a specific portion of a literary work is generally different between different literary works in the same language, the content of a specific portion of a literary work can generally be used to distinguish different literary works in the same language. Therefore, calculating the similarity between the content of specific portions of literary works in different languages ​​can be used to align literary works in different languages, thereby obtaining literary work pairs containing matching literary works.

[0177] In one embodiment of the present disclosure, the specific method of calculating the similarity between the contents of specific parts of literary works in different languages ​​can refer to the above-mentioned method of calculating the similarity between titles and the specific method of calculating the similarity between abstracts, and the details are not repeated here.

[0178] In one embodiment of the present disclosure, when a literary work in a literary work database does not include an abstract or the abstract contains too little content, it is advantageous to calculate the similarity between the content of specific portions of literary works in different languages ​​to perform literary work matching, thereby obtaining pairs of literary works in different languages, that is, to help determine whether literary works in different languages ​​are matching literary works. Of course, depending on the actual application needs, literary work matching can be performed using the calculated similarity between abstracts, the calculated similarity between specific portions of literary work content, or both the calculated similarity between abstracts and the calculated similarity between specific portions of literary work content.

[0179] In one embodiment of the present disclosure, the method of weighted calculation of the mapping relationship of titles, the calculated similarity between titles, the mapping relationship of authors, the calculated matching degree between authors, the calculated similarity between abstracts and / or the calculated similarity between specific parts of the content of literary works can refer to the above-mentioned method of calculating the similarity between titles and the specific method of calculating the similarity between abstracts, and the details are not repeated here.

[0180] In one embodiment of the present disclosure, a weighted calculation is performed on the mapping relationship of titles, the calculated similarity between titles, the mapping relationship of authors, the calculated matching degree between authors, the calculated similarity between abstracts, and / or the calculated similarity between specific parts of the content of literary works to obtain the literary works of different languages ​​as matching literary works scores in the literary works pair, and the literary works of different languages ​​are determined to be matching literary works based on the matching literary works scores exceeding the preset matching literary works score threshold, thereby obtaining a pair of literary works containing different languages. In one embodiment of the present disclosure, the mapping relationship of titles, the calculated similarity between titles, the mapping relationship of authors, the calculated matching degree between authors, the calculated similarity between abstracts, and the calculated similarity between specific parts of the content of literary works are all indicators for determining whether the literary works of different languages ​​are matching literary works. In the embodiment of the present disclosure, the more indicators are used to determine whether the literary works of different languages ​​are matching literary works, the more accurate the determination result.

[0181] In one embodiment of the present disclosure, when matching literary works are determined according to the various methods described above and pairs of literary works in different languages ​​are obtained, the mapping relationship of the titles of literary works in different languages ​​in the literary work pairs and the mapping relationship of the authors of literary works in different languages ​​in the literary work pairs can be updated. That is, the mapping relationship of the titles of literary works in the obtained pairs of literary works in different languages ​​can be updated to the "title alignment data", and the mapping relationship of the authors of literary works in the obtained pairs of literary works in different languages ​​can be updated to the "author alignment data". Therefore, for the obtained pairs of literary works in different languages, the "title alignment data" and the "author alignment data" can be updated in a closed loop. Repeat the aforementioned literary work matching (obtaining pairs of literary works in different languages) process until no new pairs of literary works in different languages ​​can be obtained.

[0182] In one embodiment of the present disclosure, step S130 includes: aligning the chapters of literary works in different languages ​​in the literary work pair according to the chapter content and chapter names of the literary works; generating sentence groups containing corresponding sentences in different languages ​​from the aligned chapters in the literary works; filtering the sentence groups based on the length ratio and / or similarity between corresponding sentences in the sentence groups according to preset sentence group filtering rules; and generating corpus pairs based on the filtered sentence groups that meet the sentence group filtering rules.

[0183] According to the technical solution provided by the embodiments of the present disclosure, corpus pairs in different languages ​​are obtained by utilizing the pairs of literary works in different languages, including: aligning the chapters of the literary works in different languages ​​in the pairs of literary works according to the chapter content and chapter names of the literary works; generating sentence groups containing corresponding sentences in different languages ​​from the aligned chapters in the literary works; screening the sentence groups based on the length ratio and / or similarity between the corresponding sentences in the sentence groups according to preset sentence group screening rules; generating corpus pairs based on the screened sentence groups that meet the sentence group screening rules, and utilizing the characteristics of literary works in different languages, combining the titles, authors, abstracts and chapter names of literary works to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0184] In one embodiment of the present disclosure, chapter content of a literary work refers to the content within each chapter of the literary work. Each chapter content is preceded by a chapter name to distinguish the current chapter content from other chapter content. The following uses a literary work pair including matching literary works in Chinese and English as an example to illustrate how to align the literary content of matching literary works in various language versions within the literary work pair to generate a corpus pair.

[0185] In one embodiment of the present disclosure, a literary work pair is defined as a parallel Chinese literary work A and an English literary work B. The corresponding chapters of Chinese literary work A are designated A_1, A_2, ..., A_m, and the corresponding chapters of English literary work B are designated B_1, B_2, ..., B_n, where m and n are positive integers. Alignment is performed on m chapters of Chinese literary work A and n chapters of English literary work B.

[0186] In one embodiment of the present disclosure, the similarity between a chapter A_i in a Chinese literary work A and a chapter B_j in an English literary work B is calculated, where i and j are positive integers. The specific calculation method is as follows: first, perform word segmentation operations on the Chinese chapter A_i and the English chapter B_j, record the Chinese word segmentation result of the Chinese chapter A_i as A_token_i, and record the English word segmentation result of the English chapter B_j as B_token_j. Thereafter, each word in the Chinese word segmentation result A_token_i is translated into English according to a Chinese-English bilingual dictionary, and recorded as the Chinese word segmentation result English translation A_token_i_translate. The number of identical n-grams in the Chinese word segmentation result English translation A_token_i_translate and the English word segmentation result B_token_j is counted. In this way, the number of identical n-grams between the Chinese chapter A_i and the English chapter B_j is counted. In this way, we can count the number of identical n-grams between a Chinese article A_i and all n articles B_1, B_2, ...B_n in the English literary work B. From these n article pairs, we filter out the top k article pairs with the highest number of identical n-grams (i.e., the highest article content similarity), where k is a positive integer, for example, 4 (n ≥ 4), i.e., the top four article pairs with the highest number of identical n-grams. In other words, we sort these n article pairs in descending order of the number of identical n-grams, and filter out the top article pairs as the highly similar article pairs.

[0187] In one embodiment of the present disclosure, the first k (for example, 4) potential English passages to be matched that may be aligned with the Chinese passage A_i are recorded as B_k1, B_k2, B_k3 and B_k4 respectively. Next, the passage number distance is scored from these 4 screening results, that is, the passage number distance is scored for the passage pair including the Chinese passage A_i and the English passage to be matched B_k1, the passage pair including the Chinese passage A_i and the English passage to be matched B_k2, the passage pair including the Chinese passage A_i and the English passage to be matched B_k3, and the passage pair including the Chinese passage A_i and the English passage to be matched B_k4. The passage number distance score means that, when each passage has a corresponding passage number, the Chinese passage number is represented in the form of "Chapter One", "Chapter 1", etc., and the English passage number is represented in the form of "Chapter 1", "Chapter one", etc. First, normalize the chapter numbers, converting them from Chinese, English, and Roman numerals to Arabic numerals. Then, calculate the absolute difference between the Chinese and English chapter numbers. For example, the absolute difference between the Chinese chapter number "Chapter 1" and the English chapter number "Chapter 1" is 0. The absolute difference between the Chinese chapter number "Chapter 15" and the English chapter number "Chapter 12" is 3.

[0188] In one embodiment of the present disclosure, a chapter name includes a chapter number and / or a chapter title. In this embodiment of the present disclosure, a chapter title refers to a word, phrase, or sentence that is located after the chapter number and together with the chapter number constitutes the chapter name, or that exists alone to constitute the chapter name. For example, a Chinese chapter name may include only a chapter number such as "Chapter One" or "Chapter 1"; it may also include only a chapter title such as "The Beginning of the Story" or "Newton's First Law"; it may also include both a chapter number and a chapter title, such as "Chapter 1 The Beginning of the Story" or "Chapter 1 Newton's First Law". For example, an English chapter name may include only a chapter number such as "Chapter 1" or "Chapter one"; it may also include only a chapter title such as "Beginning of the Story" or "Newton's First Law"; it may also include both a chapter number and a chapter title, such as "Chapter 1 Beginning of the Story" or "Chapter one Newton's First Law".

[0189] In one embodiment of the present disclosure, the chapter title similarity is calculated for high-similarity chapter pairs. For example, based on the aforementioned example, the chapter title similarity is calculated for the chapter pair including the Chinese chapter A_i and the English chapter to be matched B_k1, the chapter pair including the Chinese chapter A_i and the English chapter to be matched B_k2, the chapter pair including the Chinese chapter A_i and the English chapter to be matched B_k3, and the chapter pair including the Chinese chapter A_i and the target English chapter B_k4 to score the chapter title similarity. The method for calculating the chapter title similarity can refer to the method for calculating the similarity between titles and the specific method for calculating the similarity between abstracts, and the details are not repeated here.

[0190] In one embodiment of the present disclosure, the chapter name similarity of each chapter pair with high similarity can be obtained by performing a weighted calculation on the absolute value of the difference between the Chinese chapter number and the English chapter number and / or the chapter title similarity, thereby screening out the chapter pair with the highest chapter name similarity from the high similarity chapter pairs as the aligned chapters.

[0191] In one embodiment of the present disclosure, sentence groups can be generated from aligned chapters. Referring to the example of the above literary works including parallel Chinese literary work A and English literary work B, sentence pairs can be generated from aligned chapters. In one embodiment of the present disclosure, sentence pairs can be screened according to the length ratio of sentence pairs and / or the similarity of sentence pairs to obtain the final corpus. For example, the similarity of sentence pairs can be calculated by translating and calculating based on a mutual translation probability dictionary, and its specific calculation method can be obtained from the above discussion. For example, the similarity of sentence pairs can be calculated with reference to the above-mentioned method of calculating the similarity between titles and the specific method of calculating the similarity between abstracts, and the details are not repeated here. The length ratio of a sentence pair refers to the length ratio between corresponding sentences in a sentence group. For example, in a Chinese-English sentence pair, the ratio of the length of the Chinese sentence to the length of the English sentence should be within a certain threshold range. If the threshold range is exceeded, the Chinese sentence and the English sentence in the Chinese-English sentence pair are not aligned sentences.

[0192] In one embodiment of the present disclosure, chapters of literary works in different languages ​​in a pair of literary works are aligned according to the chapter contents and chapter names of the literary works, including: segmenting the chapter contents in different languages ​​in the pair of literary works; translating the words in the chapters of the translated literary works according to a preset multi-language translation dictionary to count the number of identical n-grams in the chapters of the translated literary works and the chapters of the literary works to be matched; sorting the chapter groups consisting of the chapters of the translated literary works and the chapters of the literary works to be matched from high to low according to the number of identical n-grams, and screening out the top k chapter groups from the sorted chapter groups as high-similarity chapter groups, where k is a positive integer; calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter groups; and screening out the chapter group with the highest similarity in chapter name from the high-similarity chapter groups as the aligned chapters.

[0193] According to the technical solution provided by the embodiment of the present disclosure, chapters of literary works in different languages ​​in a literary work pair are aligned according to the chapter content and chapter name of the literary works, including: segmenting the chapter content of different languages ​​in the literary work pair; translating the words in the chapters of the translated literary works according to a preset multi-language translation dictionary to count the number of identical n-grams in the chapters of the translated literary works and the chapters of the literary works to be matched; and aligning the chapter groups composed of the chapters of the translated literary works and the chapters of the literary works to be matched according to the number of identical n-grams. The number of am is sorted from high to low, and the first k chapter groups are selected from the sorted chapter groups as high-similarity chapter groups, where k is a positive integer; the similarity of the chapter names in different languages ​​in the high-similarity chapter groups is calculated; the chapter group with the highest similarity in the chapter name is selected from the high-similarity chapter group as the aligned chapters, and the characteristics of literary works in different languages ​​can be utilized, combined with the title, author, abstract and chapter name of the literary works to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0194] In one embodiment of the present disclosure, calculating the similarity of chapter names in different languages ​​in the high-similarity chapter group includes: normalizing the chapter numbers in the chapter names in different languages ​​in the high-similarity chapter group to calculate the absolute value of the difference between the chapter numbers in different languages; and / or calculating the similarity between the chapter titles in the chapter names in different languages ​​in the high-similarity chapter group; calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter group based on the absolute value of the difference between the chapter numbers in different languages ​​and / or the similarity between the chapter titles in the chapter names in different languages.

[0195] According to the technical solution provided by the embodiment of the present disclosure, by calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter group, including: normalizing the chapter numbers in the chapter names in different languages ​​in the high-similarity chapter group to calculate the absolute value of the difference between the chapter numbers in different languages; and / or calculating the similarity between the chapter titles in the chapter names in different languages ​​in the high-similarity chapter group; calculating the similarity of the chapter names in different languages ​​in the high-similarity chapter group based on the absolute value of the difference between the chapter numbers in different languages ​​and / or the similarity between the chapter titles in the chapter names in different languages, the characteristics of literary works in different languages ​​can be utilized, and literary works matching and chapter alignment can be performed in combination with the title, author, abstract and chapter name of the literary works, so as to automatically generate high-quality corpus pairs from literary works in different languages ​​efficiently and accurately.

[0196] The above examples of bilingual Chinese and English literary works have been used to discuss the alignment of literary works, the alignment of chapters, and the generation of sentence pairs. However, it should be understood that the examples of bilingual Chinese and English literary works can be applied to the alignment of literary works in three or more different languages, the alignment of chapters, and the generation of sentence groups. For example, corresponding processing can be performed for literary works in three or more different languages ​​when executing each step of the corpus generation method. For example, when calculating the title similarity of literary works in three languages, the literary works in three languages ​​can be divided into three literary work pairs, the title similarity can be calculated for each literary work pair, and the similarity between the titles of the literary works in three language versions can be calculated by calculating the average value of the sum of the title similarities of the three literary work pairs. For another example, the literary works in three or more different languages ​​can be split into multiple bilingual literary work pairs, and the alignment of literary works, the alignment of chapters, and the generation of sentence pairs can be performed on each bilingual literary work pair respectively, and sentence groups containing three or more languages ​​can be finally generated, thereby obtaining a final multilingual corpus.

[0197] It should be noted that the embodiments of the present disclosure adopt a series of means such as title similarity, author similarity, abstract similarity, content similarity of specific parts and comprehensive weighted scoring to produce corpus from literary works in different languages, thereby realizing a complete set of solutions from literary work matching, chapter alignment, sentence pair generation, and corpus generation, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0198] The following reference Figure 2 A corpus generation device according to another embodiment of the present disclosure is described.

[0199] Figure 2 FIG. 1 shows a structural block diagram of a corpus generation device according to an embodiment of the present disclosure. Figure 2 As shown, the corpus generation device 200 includes an acquisition module 210 , a matching module 220 and an obtaining module 230 .

[0200] The acquisition module 210 is configured to acquire literary works in different languages ​​and feature data of the literary works.

[0201] The matching module 220 is configured to match literary works in different languages ​​according to the feature data of the literary works to obtain pairs of literary works in different languages.

[0202] The obtaining module 230 is configured to obtain a corpus pair in a different language using the pair of literary works in different languages.

[0203] According to the technical solution provided by the embodiments of the present disclosure, the acquisition module is configured to acquire literary works in different languages ​​and feature data of the literary works; the matching module is configured to match the literary works in different languages ​​based on the feature data of the literary works to obtain pairs of literary works in different languages; the acquisition module is configured to use the pairs of literary works in different languages ​​to obtain corpus pairs in different languages. The characteristics of literary works in different languages ​​can be used to match the literary works in combination with the titles, authors, abstracts and chapter names of the literary works to obtain pairs of literary works in different languages, thereby efficiently and accurately automatically generating high-quality corpus pairs from literary works in different languages.

[0204] Those skilled in the art will understand that referring to Figure 2 The technical solutions described can be compared with Figure 1 The embodiments described are combined to provide a reference Figure 1 The technical effects achieved by the described embodiments can be referred to above for details. Figure 1 The specific content will not be repeated here.

[0205] The following reference Figure 3 An exemplary application scenario of generating corpus pairs using a bilingual version of a novel by using the corpus generation method according to an embodiment of the present disclosure is described.

[0206] Figure 3 A schematic diagram illustrating an exemplary application of generating a corpus pair using a bilingual version of a novel using the corpus generation method according to an embodiment of the present disclosure.

[0207] 1. The acquired novel data may include the novel title, author name, chapter number, chapter title, and the literary content of the chapter. Currently, the acquired novels are available in both Chinese and English. Novels in these two languages ​​can be stored in two databases: the "Chinese Novels" database and the "English Novels" database.

[0208] 2. The title mapping relationship between Chinese and English (such as the mapping relationship between the novel "Journey to the West" and the English name "Journey to the West") and the author name mapping relationship between Chinese and English (such as the mapping relationship between the author "Wu Cheng'en" and "WU Chengen") of the novel can be obtained through the Internet, and are recorded in the "novel name alignment data" and "author name alignment data" in Figure 3 The "novel name alignment data" and "author name alignment data" constitute the novel alignment data as a whole.

[0209] 3. Select novels A (Chinese) and novels B (English) from the "Chinese novel" library and the "English novel" library respectively to perform parallel novel alignment (literary work matching), as follows:

[0210] 3.1 Chinese and English novel name pair matching score: if the novel name pair of A and B appears in the "novel name alignment data", a novel name matching score is obtained, and this score is a configurable value, such as 1, denoted as title_match_score.

[0211] 3.2 Novel name pair similarity calculation score: if the novel name pair of A and B does not appear in the "novel name alignment data", a similarity calculation score is calculated for the novel name. The similarity calculation score here comes from two scores: the first score is based on the word mutual translation probability score, which is calculated by first segmenting the Chinese and English novel names, and then calculating the similarity score title_similarity_score_1 between the two novel names through a pre-set mutual translation probability dictionary (i.e. the translation probability between two words in Chinese and English, which is derived from word alignment statistical data of parallel corpus). The second score is derived from machine translation results, which is calculated by first translating the name of A into English through a machine translation engine, denoted as A_1, and then performing English segmentation on A_1 and B, and calculating the second similarity score title_similarity_score_2 based on the edit distance and n-gram overlap between A_1 and B. The weights of title_similarity_score_1 and title_similarity_score_2 are set to obtain the novel name similarity score title_similarity_score.

[0212] 3.3 Chinese and English author name pair matching score: if the author name pair of A and B appears in the "author name alignment data", an author name matching score is obtained, denoted as author_match_score.

[0213] 3.4 Matching score between the Chinese author name converted to pinyin and the English author name: Some authors' English names are directly taken from the pinyin of their Chinese names. Therefore, if the name pair of novel authors A and B does not appear in 3.3 (i.e., not in the "author name alignment data"), the pinyin of the Chinese author name of the novel is obtained. If the pinyin obtained for A is consistent with the author name of B, the matching score of the Chinese author name converted to pinyin is obtained, which is recorded as author_pinyin_match_score.

[0214] 3.5 Calculation of similarity scores between Chinese and English novel summaries: For the novel information of A and B, the novel summaries (or description information) can usually be obtained. By using the similarity calculation method similar to the novel name pairs in 3.2, the novel summary similarity score can be obtained, which is recorded as abstract_similarity_score.

[0215] 3.6 Similarity Calculation Score for the First Chapter of Chinese and English Novels: Because some novels lack detailed descriptions, we can further determine whether the novels are aligned by calculating the similarity of the first chapter. The specific calculation method is similar to the calculation method for novel title pair similarity in 3.2. The similarity calculation score for the first chapter of a novel is denoted as first_chapter_similarity_score.

[0216] 3.7 The total score for whether A and B are parallel novels is calculated by weighting the "matching score of the Chinese and English novel titles," "the similarity calculation score of the novel titles," "the matching score of the Chinese and English author names," "the matching score of the Chinese author name converted to pinyin and then matched with the English name," "the similarity calculation score of the Chinese and English novel summaries," and "the similarity calculation score of the first chapters of the Chinese and English novels." A comparison threshold is then used to determine whether A and B are parallel novels, i.e., matching literary works.

[0217] 4. For the aligned novels obtained in step 3, update the "novel name alignment data" and "author name alignment data" in a closed loop. Repeat the novel alignment process in step 3 until no new novel pairs are generated.

[0218] 5. Based on the parallel novel pairs obtained in step 3, begin aligning the novel chapters. The specific steps are as follows:

[0219] 5.1 Let A and B represent two parallel novels, with A being the Chinese version and B being the English version. Place all chapter data corresponding to A and B into a single bucket. Assume that all chapters corresponding to A are A_1, A_2, ...A_m, and all chapters corresponding to B are B_1, B_2, ...B_n. Align m chapters in A with n chapters in B.

[0220] 5.2 Calculate the similarity between an article A_i corresponding to A and an article B_j corresponding to B. The specific method is as follows: First, perform a word segmentation operation on A_i and B_j, denoted as A_token_i and B_token_j. Then, translate each word (Chinese) in A_token_i into English according to the bilingual dictionary, denoted as A_token_i_translate, and then count the number of identical n-grams in A_token_i_translate and B_token_j (here n is also a parameter, for example, set to 2). In this way, the number of identical n-grams between A_i and B_j is counted. In this way, the number of identical n-grams in all pairs of articles A_i and B_1, B_2...B_n can be counted. Based on the number of identical n-grams, the top k article pairs with the highest similarity are selected (k is a parameter, for example, set to 4).

[0221] 5.3 The first k (assuming k is 4) potential chapters to be aligned for each chapter A_i are denoted as B_k1, B_k2, B_k3, and B_k4. Next, the chapter number distance score is calculated based on the first four results: each chapter has a corresponding chapter number. Chinese chapter numbers are represented as "章一" (Chapter 1), "第1章" (Chapter 1), etc., and English chapter numbers are represented as "Chapter 1", "Chapter one", etc. First, normalization is performed to convert Chinese, English, and Roman numerals to Arabic numerals. Then, the absolute value of the difference between the Chinese and English chapter numbers is calculated.

[0222] 5.4 Score the chapter title similarity for the first four chapter pairs corresponding to A_i. The score calculation method here refers to the calculation method of title similarity in 3.2.

[0223] 5.5 Combined with the scores of 5.3 and 5.4, the final score of each pair of passages can be obtained after weighting, and the highest value can be selected.

[0224] 6. Produce sentence pairs from aligned texts.

[0225] 7. Sentence pairs are filtered based on their length ratios and similarity scores (e.g., based on a mutual translation probability dictionary) to obtain the final bilingual novel corpus.

[0226] The following reference Figure 4 A corpus generation method according to another embodiment of the present disclosure is described.

[0227] Figure 4 FIG. 1 is a flow chart showing a method for generating corpus according to another embodiment of the present disclosure. Figure 4 As shown, the corpus generation method includes the following steps S410, S420, S430, S440, S450, S460 and S470:

[0228] In step S410, first language literary works, first language literary work feature data including the title, author, abstract and chapter name of the first language literary work, second language literary works, and second language literary work feature data including the title, author, abstract and chapter name of the second language literary work are obtained.

[0229] In step S420 , a mapping relationship between the titles of the first language literary works and the titles of the second language literary works, as well as a mapping relationship between the authors, is obtained.

[0230] In step S430 , the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts are calculated.

[0231] In step S440, a weighted calculation is performed on the mapping relationship between the titles of the first language literary work and the second language literary work, the mapping relationship between the authors, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts to obtain a literary work pair including the first language literary work and the second language literary work.

[0232] In step S450 , the chapters in the first language literary work and the second language literary work in the literary work pair are aligned according to the chapter contents and chapter names in the first language literary work and the second language literary work.

[0233] In step S460 , sentence pairs including corresponding sentences in the first language and sentences in the second language are obtained from the aligned passages.

[0234] In step S470 , sentence pairs are screened according to preset corpus screening rules to generate corpus pairs.

[0235] According to the technical solution provided by the embodiment of the present disclosure, by obtaining first language literary works, first language literary works feature data including the title, author, abstract and chapter name of the first language literary works, second language literary works and second language literary works feature data including the title, author, abstract and chapter name of the second language literary works; obtaining the mapping relationship between the title of the first language literary works and the title of the second language literary works and the mapping relationship between the authors; calculating the similarity between the titles of the first language literary works and the second language literary works, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts; and comparing the mapping relationship between the titles of the first language literary works and the second language literary works, the mapping relationship between the authors, the similarity between the titles and the similarity between the contents of specific parts. The method performs weighted calculation on the matching degree, the matching degree between authors, the similarity between abstracts and the similarity between the contents of specific parts to obtain literary work pairs containing first-language literary works and second-language literary works; aligns the chapters in the first-language literary works and second-language literary works according to the chapter contents and chapter names in the literary work pairs; obtains sentence pairs containing corresponding first-language sentences and second-language sentences from the aligned chapters; screens the sentence pairs according to preset corpus screening rules to generate corpus pairs, and can utilize the characteristics of the literary works in the two languages ​​and combine the titles, authors, abstracts and chapter names of the literary works to match the literary works and align the chapters, thereby automatically generating high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0236] In one embodiment of the present disclosure, calculating the similarity between the titles of a first-language literary work and a second-language literary work includes: calculating a first similarity score for the similarity between the titles of the first-language literary work and the titles of the second-language literary work according to a preset first-language and second-language translation dictionary; translating the titles of the first-language literary work according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the title of the second-language literary work to be matched, and calculating a second similarity score for the similarity between the titles of the first-language literary work and the titles of the second-language literary work; and performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the titles of the first-language literary work and the second-language literary work.

[0237] According to the technical solution provided by the embodiment of the present disclosure, by calculating the similarity between the titles of literary works in a first language and literary works in a second language, the method includes: calculating a first similarity score of the similarity between the titles of the literary works in the first language and the titles of the literary works in the second language according to a preset mutual translation dictionary between the first language and the second language; translating the titles of the literary works in the first language according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the title of the second language literary work to be matched, and calculating a second similarity score of the similarity between the titles of the literary works in the first language and the titles of the literary works in the second language; performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the titles of the literary works in the first language and the titles of the literary works in the second language, and utilizing the characteristics of the literary works in the two languages, combining the titles, authors, abstracts and chapter names of the literary works to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from the literary works in the two languages.

[0238] In one embodiment of the present disclosure, calculating the degree of match between the authors of a first-language literary work and a second-language literary work includes: transliterating the author of the first-language literary work according to a preset phonetic notation scheme to obtain a transliterated author name; and calculating the degree of match between the authors of the first-language literary work and the second-language literary work based on whether the transliterated author name is consistent with the author name of the second-language literary work to be matched.

[0239] According to the technical solution provided by the embodiments of the present disclosure, by calculating the matching degree between the authors of a first-language literary work and a second-language literary work, including: transliterating the author of the first-language literary work according to a preset phonetic notation scheme to obtain a transliterated author name; calculating the matching degree between the authors of the first-language literary work and the second-language literary work based on whether the transliterated author name is consistent with the author name of the second-language literary work to be matched, the characteristics of the literary works in the two languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, thereby efficiently and accurately automatically generating high-quality corpus pairs from the literary works in the two languages.

[0240] In one embodiment of the present disclosure, sentence pairs are screened according to preset corpus screening rules to generate corpus pairs, including: screening sentence pairs based on the length ratio and / or similarity between corresponding sentences in the sentence pairs according to the preset sentence pair screening rules; and generating corpus pairs based on the screened sentence pairs that meet the sentence pair screening rules.

[0241] According to the technical solution provided by the embodiments of the present disclosure, sentence pairs are screened according to preset corpus screening rules to generate corpus pairs, including: according to the preset sentence pair screening rules, sentence pairs are screened based on the length ratio and / or similarity between corresponding sentences in the sentence pairs; corpus pairs are generated according to the screened sentence pairs that meet the sentence pair screening rules. The characteristics of literary works in two languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from literary works in two languages ​​efficiently and accurately.

[0242] In one embodiment of the present disclosure, chapters in a first-language literary work and a second-language literary work in a literary work pair are aligned based on chapter contents and chapter names in the first-language literary work and the second-language literary work in the literary work pair, including: segmenting the chapter contents of the first-language literary work and the second-language literary work in the literary work pair; translating words in the chapters of the first-language literary work according to a preset first-language and second-language translation dictionary to count the number of identical n-grams in the chapters of the first-language literary work and the chapters of the second-language literary work; sorting the chapter pairs consisting of the chapters of the first-language literary work and the chapters of the second-language literary work from high to low according to the number of identical n-grams, and selecting the top k chapter pairs from the sorted chapter pairs as high-similarity chapter pairs, where k is a positive integer; calculating the similarity between the chapter names in the first language and the chapter names in the high-similarity chapter pairs; and selecting the chapter pair with the highest similarity in chapter name from the high-similarity chapter pairs as the aligned chapters.

[0243] According to the technical scheme provided by the embodiment of the present disclosure, the chapters in the first language literary work and the second language literary work are aligned according to the chapter content and the chapter name in the first language literary work and the second language literary work in the literary work pair, including: segmenting the chapter content of the first language literary work and the second language literary work in the literary work pair; translating the words in the chapter of the first language literary work according to the preset first language and second language mutual translation dictionary to count the number of same n-grams in the chapter of the first language literary work and the chapter of the second language literary work; sorting the chapter pairs composed of the chapter of the first language literary work and the chapter of the second language literary work according to the number of same n-grams from high to low, and selecting the top k chapter pairs as the high-similarity chapter pairs from the sorted chapter pairs, wherein k is a positive integer; calculating the similarity of the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair; selecting the chapter pair with the highest similarity of the chapter name from the high-similarity chapter pair as the aligned chapter, which can utilize the characteristics of literary works in two languages, and combine the title, author, abstract and chapter name of the literary work to match the literary works and align the chapter pairs, thereby efficiently and accurately generating high-quality corpus pairs from two language literary works.

[0244] In one embodiment of the present disclosure, the similarity of the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair is calculated, including: normalizing the chapter numbers in the chapter name in the first language and the chapter numbers in the chapter name in the second language in the high-similarity chapter pair to calculate the absolute value of the difference between the chapter numbers in the chapter name in the first language and the chapter numbers in the chapter name in the second language; and / or calculating the similarity between the chapter names in the chapter name in the first language and the chapter names in the chapter name in the second language in the high-similarity chapter pair; calculating the similarity of the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair according to the absolute value of the difference between the chapter numbers in the chapter name in the first language and the chapter numbers in the chapter name in the second language and / or the similarity between the chapter names in the chapter name in the first language and the chapter names in the chapter name in the second language.

[0245] According to the technical solution provided by the embodiment of the present disclosure, by calculating the similarity between the chapter name in the first language and the chapter name in the second language in the high similarity chapter pair, the method includes: normalizing the chapter number in the chapter name in the first language and the chapter number in the chapter name in the second language in the high similarity chapter pair to calculate the absolute value of the difference between the chapter number in the chapter name in the first language and the chapter number in the chapter name in the second language; and / or calculating the similarity between the chapter title in the chapter name in the first language and the chapter title in the chapter name in the second language in the high similarity chapter pair; based on Based on the absolute value of the difference between the chapter number in the chapter name of the first language and the chapter number in the chapter name of the second language and / or the similarity between the chapter label in the chapter name of the first language and the chapter label in the chapter name of the second language, the similarity between the chapter name in the first language and the chapter name in the second language in a high-similarity chapter pair is calculated. The characteristics of the literary works in the two languages ​​can be utilized, and the titles, authors, abstracts and chapter names of the literary works can be combined to perform literary work matching and chapter alignment, so as to automatically generate high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0246] Those skilled in the art will understand that referring to Figure 4 The technical solutions described can be compared with Figures 1 to 3 The embodiments described are combined to provide a reference Figures 1 to 3 The technical effects achieved by the described embodiments can be referred to above for details. Figures 1 to 3 The specific content will not be repeated here.

[0247] The following reference Figure 5 A corpus generation device according to another embodiment of the present disclosure is described.

[0248] Figure 5 FIG. 5 shows a structural block diagram of a corpus generating device 500 according to another embodiment of the present disclosure. Figure 5 As shown, the corpus generation device 500 includes a first acquisition module 510 , a second acquisition module 520 , a first calculation module 530 , a second calculation module 540 , an alignment module 550 , a third acquisition module 560 and a generation module 570 .

[0249] The first acquisition module 510 is configured to acquire first language literary works, first language literary work feature data including the title, author, abstract and chapter name of the first language literary works, second language literary works and second language literary work feature data including the title, author, abstract and chapter name of the second language literary works.

[0250] The second acquisition module 520 is configured to acquire a mapping relationship between titles of first-language literary works and titles of second-language literary works, as well as a mapping relationship between authors.

[0251] The first calculation module 530 is configured to calculate the similarity between the titles of the literary works in the first language and the literary works in the second language, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts.

[0252] The second calculation module 540 is configured to perform weighted calculation on the mapping relationship between the titles of the literary works in the first language and the literary works in the second language, the mapping relationship between the authors, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts, and obtain a literary work pair containing the literary works in the first language and the literary works in the second language.

[0253] The alignment module 550 is configured to align the chapters in the literary works in the first language and the literary works in the second language according to the chapter contents and chapter names in the literary works in the first language and the literary works in the second language in the literary work pair.

[0254] The third obtaining module 560 is configured to obtain, from the aligned chapters, a sentence pair containing the first language sentences and the second language sentences corresponding to each other.

[0255] The generation module 570 is configured to perform screening on the sentence pair according to a preset corpus screening rule to generate a corpus pair.

[0256] According to the technical solution provided by the embodiment of the present disclosure, a first acquisition module is configured to acquire first language literary works, first language literary work feature data including the title, author, abstract and chapter name of the first language literary work, and second language literary works, and second language literary work feature data including the title, author, abstract and chapter name of the second language literary work; a second acquisition module is configured to acquire the mapping relationship between the title of the first language literary work and the title of the second language literary work, and the mapping relationship between the authors; a first calculation module is configured to calculate the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts; a second calculation module is configured to calculate the mapping relationship between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts; The method further comprises the following steps: performing weighted calculation on the mapping relationship between the two, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts and the similarity between the contents of specific parts to obtain a literary work pair containing a first-language literary work and a second-language literary work; an alignment module is configured to align the chapters in the first-language literary work and the second-language literary work in the literary work pair according to the chapter contents and chapter names in the first-language literary work and the second-language literary work; a third acquisition module is configured to obtain sentence pairs containing corresponding sentences in the first language and sentences in the second language from the aligned chapters; a generation module is configured to screen the sentence pairs according to preset corpus screening rules to generate corpus pairs, which can utilize the characteristics of the literary works in the two languages ​​and combine the titles, authors, abstracts and chapter names of the literary works to perform literary work matching and chapter alignment, thereby automatically generating high-quality corpus pairs from the literary works in the two languages ​​efficiently and accurately.

[0257] Those skilled in the art will understand that referring to Figure 5 The technical solutions described can be compared with Figures 1 to 4 The embodiments described are combined to provide a reference Figures 1 to 4 The technical effects achieved by the described embodiments can be referred to above for details. Figures 1 to 4 The specific content will not be repeated here.

[0258] The above embodiments describe the internal functions and structures of the corpus generation device. In a possible design, the structure of the corpus generation device can be implemented as an electronic device, such as Figure 6 As shown in , the electronic device 600 may include a processor 601 and a memory 602 .

[0259] The memory 602 is used to store a program that supports the electronic device to execute the corpus generation method or code generation method in any of the above embodiments, and the processor 601 is configured to execute the program stored in the memory 602.

[0260] In one embodiment of the present disclosure, the memory 602 is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 601 to implement the following steps:

[0261] Obtain literary works in different languages ​​and characteristic data of literary works;

[0262] matching literary works in different languages ​​according to the characteristic data of the literary works to obtain pairs of literary works in different languages;

[0263] Corpus pairs in different languages ​​are obtained by using the pairs of literary works in different languages.

[0264] In one embodiment of the present disclosure, the characteristic data of the literary work includes one or more of the title, author, abstract, and chapter name of the literary work.

[0265] In one embodiment of the present disclosure, matching literary works in different languages ​​according to the feature data of the literary works to obtain pairs of literary works in different languages ​​includes:

[0266] Obtain the mapping relationship between the titles of literary works in different languages;

[0267] Obtain mapping relationships between authors of literary works in different languages;

[0268] Literary works in different languages ​​are matched according to the mapping relationship between the titles and the mapping relationship between the authors.

[0269] In one embodiment of the present disclosure, the method further includes:

[0270] Calculate the similarity between titles of literary works in different languages;

[0271] Calculate the matching degree between authors of literary works in different languages;

[0272] The mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, and the matching degree between the authors are weightedly calculated in a preset manner to obtain pairs of literary works in different languages.

[0273] In one embodiment of the present disclosure, the calculating the similarity between the titles of literary works in different languages ​​includes:

[0274] Calculating a first similarity score of similarity between titles of literary works in different languages ​​according to a preset multi-language translation dictionary;

[0275] Translate the titles of literary works in different languages ​​according to a preset translation tool to obtain translation results;

[0276] Perform word segmentation and overlap calculation on the translation results and the titles of the literary works to be matched, and calculate a second similarity score of the similarity between the titles of the literary works in different languages;

[0277] A weighted calculation is performed on the first similarity score and the second similarity score to obtain the similarity between the titles of literary works in different languages.

[0278] In one embodiment of the present disclosure, calculating the matching degree between authors of literary works in different languages ​​includes:

[0279] Transliterating the authors of literary works in different languages ​​according to a preset phonetic scheme to obtain transliterated author names;

[0280] The matching degree between the authors of the literary works in different languages ​​is calculated based on whether the transliterated author name is consistent with the author name of the literary work to be matched.

[0281] In one embodiment of the present disclosure, the method further includes:

[0282] Computing similarities between abstracts of literary works in different languages; and / or

[0283] Calculate the similarity between the content of specific parts of literary works in different languages;

[0284] A weighted calculation is performed on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, the matching degree between the authors, the similarity between the abstracts, and / or the similarity between the contents of specific parts to obtain pairs of literary works in different languages.

[0285] In one embodiment of the present disclosure, the step of obtaining a corpus pair in different languages ​​by using the pair of literary works in different languages ​​includes:

[0286] aligning chapters of literary works in different languages ​​in a pair of literary works according to chapter contents and chapter names of the literary works;

[0287] Generate sentence groups containing corresponding sentences in different languages ​​from aligned passages in literary works;

[0288] According to a preset sentence group screening rule, the sentence groups are screened based on the length ratio and / or similarity between corresponding sentences in the sentence groups;

[0289] Corpus pairs are generated based on the filtered sentence groups that meet the sentence group screening rules.

[0290] In one embodiment of the present disclosure, aligning the chapters of the literary works in different languages ​​in the literary work pair according to the chapter contents and chapter names of the literary works includes:

[0291] Segment the content of different languages ​​in literary works;

[0292] Translate words in the chapter of the translated literary work according to a preset multi-language translation dictionary to count the number of identical n-grams between the chapter of the translated literary work and the chapter of the literary work to be matched;

[0293] Sorting the chapter groups consisting of the chapters of the translated literary work and the chapters of the literary work to be matched from high to low according to the number of identical n-grams, and selecting the top k chapter groups from the sorted chapter groups as high similarity chapter groups, where k is a positive integer;

[0294] Calculating the similarity of chapter titles in different languages ​​in the high-similarity chapter group;

[0295] The chapter group with the highest similarity in chapter name is selected from the high-similarity chapter groups as the aligned chapters.

[0296] In one embodiment of the present disclosure, the calculating of the similarity of chapter names in different languages ​​in the high-similarity chapter group includes:

[0297] Normalizing the chapter numbers in the chapter names of different languages ​​in the high-similarity chapter group to calculate the absolute value of the difference between the chapter numbers in different languages; and / or

[0298] Calculating the similarity between the chapter titles in the chapter names of different languages ​​in the high-similarity chapter group;

[0299] The similarity of the chapter names in different languages ​​in the high-similarity chapter group is calculated according to the absolute value of the difference between the chapter numbers in different languages ​​and / or the similarity between the chapter titles in the chapter names in different languages.

[0300] In one embodiment of the present disclosure, the memory 602 is configured to store one or more computer instructions, wherein the one or more computer instructions are further executed by the processor 601 to implement the following steps:

[0301] Obtaining first-language literary works, first-language literary work feature data including the title, author, abstract, and chapter names of the first-language literary works, and second-language literary works, second-language literary work feature data including the title, author, abstract, and chapter names of the second-language literary works;

[0302] Obtaining a mapping relationship between the titles of the first language literary works and the titles of the second language literary works, as well as a mapping relationship between the authors;

[0303] Calculating the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts;

[0304] performing a weighted calculation on the mapping relationship between the titles of the first language literary work and the second language literary work, the mapping relationship between the authors, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts to obtain a literary work pair including the first language literary work and the second language literary work;

[0305] Aligning the chapters in the first-language literary work and the second-language literary work according to the chapter content and chapter names in the first-language literary work and the second-language literary work in the literary work pair;

[0306] Obtain sentence pairs containing corresponding first language sentences and second language sentences from the aligned texts;

[0307] The sentence pairs are screened according to preset corpus screening rules to generate corpus pairs.

[0308] In one embodiment of the present disclosure, calculating the similarity between the titles of a first language literary work and a second language literary work includes:

[0309] Calculating a first similarity score for the similarity between the title of the literary work in the first language and the title of the literary work in the second language according to a preset mutual translation dictionary between the first language and the second language;

[0310] Translate the title of the first language literary work according to a preset translation tool to obtain a translation result;

[0311] Performing word segmentation and overlap calculation on the translation result and the title of the second language literary work to be matched, and calculating a second similarity score of the similarity between the title of the first language literary work and the title of the second language literary work;

[0312] A weighted calculation is performed on the first similarity score and the second similarity score to obtain the similarity between the title of the literary work in the first language and the title of the literary work in the second language.

[0313] In one embodiment of the present disclosure, calculating the degree of matching between the authors of the first language literary work and the second language literary work includes:

[0314] Transliterate the author of the first language literary work according to a preset phonetic scheme to obtain the transliterated author name;

[0315] The matching degree between the authors of the first language literary work and the second language literary work is calculated based on whether the transliterated author name is consistent with the author name of the second language literary work to be matched.

[0316] In one embodiment of the present disclosure, screening the sentence pairs according to a preset corpus screening rule to generate corpus pairs includes:

[0317] According to a preset sentence pair screening rule, the sentence pairs are screened based on the length ratio and / or similarity between corresponding sentences in the sentence pairs;

[0318] Corpus pairs are generated based on the filtered sentence pairs that meet the sentence pair screening rules.

[0319] In one embodiment of the present disclosure, aligning chapters in the first language literary work and the second language literary work in the literary work pair according to chapter content and chapter names in the first language literary work and the second language literary work includes:

[0320] Segment the text content of the first language literary work and the second language literary work in the literary work pair;

[0321] translating words in a passage of the first language literary work according to a preset mutual translation dictionary between the first language and the second language to count the number of identical n-grams in the passage of the first language literary work and the passage of the second language literary work;

[0322] Sorting the passage pairs consisting of the passages of the first language literary work and the passages of the second language literary work from high to low according to the number of identical n-grams, and selecting the top k passage pairs from the sorted passage pairs as the high-similarity passage pairs, where k is a positive integer;

[0323] Calculating the similarity between the chapter title in the first language and the chapter title in the second language in the high-similarity chapter pair;

[0324] The chapter pairs with the highest similarity in chapter name are selected from the high-similarity chapter pairs as aligned chapters.

[0325] In one embodiment of the present disclosure, the calculating the similarity of the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair comprises:

[0326] normalizing the chapter numbers in the chapter name in the first language and the chapter numbers in the chapter name in the second language in the high-similarity chapter pair to calculate the absolute value of the difference between the chapter numbers in the chapter name in the first language and the chapter numbers in the chapter name in the second language; and / or

[0327] calculating the similarity between the chapter titles in the chapter name in the first language and the chapter titles in the chapter name in the second language in the high-similarity chapter pair;

[0328] According to the absolute value of the difference between the chapter numbers in the chapter name in the first language and the chapter numbers in the chapter name in the second language and / or the similarity between the chapter titles in the chapter name in the first language and the chapter titles in the chapter name in the second language, the similarity of the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair is calculated.

[0329] The exemplary embodiments of the present disclosure also provide a computer storage medium for storing computer software instructions for the positioning device, which contains programs for executing any of the above-mentioned embodiments, thereby having the technical effects of the method.

[0330] Figure 7 is a structural schematic diagram of a computer system suitable for implementing the corpus generation method according to an embodiment of the present disclosure.

[0331] As shown in Figure 7 , the computer system 700 includes a processor (CPU, TPU, GPU, FPGA, etc.) 701, which can perform various processes in the above-mentioned embodiments according to the programs stored in the read-only memory (ROM) 702 or the programs loaded from the storage portion 708 into the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0332] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.

[0333] In particular, according to embodiments of the present disclosure, the methods described above with reference to the accompanying drawings can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for executing the methods described in the accompanying drawings. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 709 and / or installed from the removable media 711.

[0334] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the diagram or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0335] The units or modules described in the embodiments of the present disclosure may be implemented in software or hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, limit the units or modules themselves.

[0336] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be included in the apparatus described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the apparatus. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the method described in the present disclosure, thereby achieving the technical effects of the method.

[0337] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

Claims

1. A corpus generation method, characterized in that: include: Obtain literary works in different languages ​​and characteristic data of literary works; matching literary works in different languages ​​according to the characteristic data of the literary works to obtain pairs of literary works in different languages; Using the pairs of literary works in different languages, obtaining corpus pairs in different languages; The method further includes: calculating a first similarity score of the similarity between the titles of literary works in different languages ​​based on a preset translation dictionary of multiple languages; translating the titles of literary works in different languages ​​based on a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the titles of the literary works to be matched, and calculating a second similarity score of the similarity between the titles of literary works in different languages; and performing a weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the titles of literary works in different languages.

2. The method according to claim 1, characterized in that The characteristic data of the literary work includes: one or more of the title, author, abstract, and chapter name of the literary work.

3. The method according to claim 2, characterized in that The matching of literary works in different languages ​​according to the feature data of the literary works to obtain pairs of literary works in different languages ​​includes: Obtain the mapping relationship between the titles of literary works in different languages; Obtain mapping relationships between authors of literary works in different languages; Literary works in different languages ​​are matched according to the mapping relationship between the titles and the mapping relationship between the authors.

4. The method according to claim 3, characterized in that The method further comprises: Calculate the matching degree between authors of literary works in different languages; The mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, and the matching degree between the authors are weightedly calculated in a preset manner to obtain pairs of literary works in different languages.

5. The method according to claim 4, characterized in that The calculation of the matching degree between authors of literary works in different languages ​​includes: Transliterating the authors of literary works in different languages ​​according to a preset phonetic scheme to obtain transliterated author names; The matching degree between the authors of the literary works in different languages ​​is calculated based on whether the transliterated author name is consistent with the author name of the literary work to be matched.

6. The method according to claim 4, characterized in that The method further comprises: Computing similarities between abstracts of literary works in different languages; and / or Calculate the similarity between the content of specific parts of literary works in different languages; A weighted calculation is performed on the mapping relationship between the titles, the similarity between the titles, the mapping relationship between the authors, the matching degree between the authors, the similarity between the abstracts, and / or the similarity between the contents of specific parts to obtain pairs of literary works in different languages.

7. The method according to claim 2, characterized in that The step of obtaining corpus pairs in different languages ​​by using the pairs of literary works in different languages ​​includes: aligning chapters of literary works in different languages ​​in a pair of literary works according to chapter contents and chapter names of the literary works; Generate sentence groups containing corresponding sentences in different languages ​​from aligned passages in literary works; According to a preset sentence group screening rule, the sentence groups are screened based on the length ratio and / or similarity between corresponding sentences in the sentence groups; Corpus pairs are generated based on the filtered sentence groups that meet the sentence group screening rules.

8. The method according to claim 7, characterized in that The aligning of the chapters of the literary works in different languages ​​in the literary work pair according to the chapter contents and chapter names of the literary works includes: Segment the content of different languages ​​in literary works; Translate words in the chapter of the translated literary work according to a preset multi-language translation dictionary to count the number of identical n-grams between the chapter of the translated literary work and the chapter of the literary work to be matched; Sorting the chapter groups consisting of the chapters of the translated literary work and the chapters of the literary work to be matched from high to low according to the number of identical n-grams, and selecting the top k chapter groups from the sorted chapter groups as high similarity chapter groups, where k is a positive integer; Calculating the similarity of chapter titles in different languages ​​in the high-similarity chapter group; The chapter group with the highest similarity in chapter name is selected from the high-similarity chapter groups as the aligned chapters.

9. The method according to claim 8, characterized in that Calculating the similarity of chapter titles in different languages ​​in the high-similarity chapter group includes: Normalizing the chapter numbers in the chapter names of different languages ​​in the high-similarity chapter group to calculate the absolute value of the difference between the chapter numbers in different languages; and / or Calculating the similarity between the chapter titles in the chapter names of different languages ​​in the high-similarity chapter group; The similarity of the chapter names in different languages ​​in the high-similarity chapter group is calculated according to the absolute value of the difference between the chapter numbers in different languages ​​and / or the similarity between the chapter titles in the chapter names in different languages.

10. A corpus generation device, characterized in that: include: an acquisition module configured to acquire literary works in different languages ​​and feature data of the literary works; a matching module configured to match literary works in different languages ​​according to the feature data of the literary works to obtain pairs of literary works in different languages; an acquisition module configured to obtain corpus pairs in different languages ​​using the pairs of literary works in different languages; The device further includes: a calculation module configured to: calculate a first similarity score of the similarity between the titles of literary works in different languages ​​according to a preset translation dictionary of multiple languages; translate the titles of literary works in different languages ​​according to a preset translation tool to obtain a translation result; perform word segmentation and overlap calculation on the translation result and the title of the literary work to be matched, and calculate a second similarity score of the similarity between the titles of literary works in different languages; and perform weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the titles of literary works in different languages.

11. A corpus generation method, characterized in that: include: Obtaining first-language literary works, first-language literary work feature data including the title, author, abstract, and chapter names of the first-language literary works, and second-language literary works, second-language literary work feature data including the title, author, abstract, and chapter names of the second-language literary works; Obtaining a mapping relationship between the titles of the first language literary works and the titles of the second language literary works, as well as a mapping relationship between the authors; Calculating the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts; performing a weighted calculation on the mapping relationship between the titles of the first language literary work and the second language literary work, the mapping relationship between the authors, the similarity between the titles, the matching degree between the authors, the similarity between the abstracts, and the similarity between the contents of specific parts to obtain a literary work pair including the first language literary work and the second language literary work; Aligning the chapters in the first-language literary work and the second-language literary work according to the chapter content and chapter names in the first-language literary work and the second-language literary work in the literary work pair; Obtain sentence pairs containing corresponding first language sentences and second language sentences from the aligned texts; Screening the sentence pairs according to preset corpus screening rules to generate corpus pairs; The method further includes: calculating a first similarity score for the similarity between the title of the first language literary work and the title of the second language literary work according to a preset mutual translation dictionary between the first language and the second language; translating the title of the first language literary work according to a preset translation tool to obtain a translation result; performing word segmentation and overlap calculation on the translation result and the title of the second language literary work to be matched, and calculating a second similarity score for the similarity between the title of the first language literary work and the title of the second language literary work; and performing weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the title of the first language literary work and the title of the second language literary work.

12. The method according to claim 11, characterized in that Calculating the matching degree between the authors of the first language literary work and the second language literary work includes: Transliterate the author of the first language literary work according to a preset phonetic scheme to obtain the transliterated author name; The matching degree between the authors of the first language literary work and the second language literary work is calculated based on whether the transliterated author name is consistent with the author name of the second language literary work to be matched.

13. The method according to claim 11, characterized in that The step of screening the sentence pairs according to the preset corpus screening rules to generate corpus pairs includes: According to a preset sentence pair screening rule, the sentence pairs are screened based on the length ratio and / or similarity between corresponding sentences in the sentence pairs; Corpus pairs are generated based on the filtered sentence pairs that meet the sentence pair screening rules.

14. The method according to claim 11, characterized in that The aligning of chapters in the first language literary work and the second language literary work according to chapter contents and chapter names in the first language literary work and the second language literary work in the literary work pair includes: Segment the text content of the first language literary work and the second language literary work in the literary work pair; translating words in a passage of the first language literary work according to a preset mutual translation dictionary between the first language and the second language to count the number of identical n-grams in the passage of the first language literary work and the passage of the second language literary work; Sorting the passage pairs consisting of the passages of the first language literary work and the passages of the second language literary work from high to low according to the number of identical n-grams, and selecting the top k passage pairs from the sorted passage pairs as the high-similarity passage pairs, where k is a positive integer; Calculating the similarity between the chapter title in the first language and the chapter title in the second language in the high-similarity chapter pair; The chapter pairs with the highest similarity in chapter name are selected from the high-similarity chapter pairs as aligned chapters.

15. The method according to claim 14, characterized in that Calculating the similarity between the chapter title in the first language and the chapter title in the second language in the high-similarity chapter pair includes: Normalizing the chapter numbers in the first language chapter name and the second language chapter name in the high-similarity chapter pair to calculate the absolute value of the difference between the chapter numbers in the first language chapter name and the chapter numbers in the second language chapter name; and / or Calculating the similarity between the chapter title in the first language and the chapter title in the second language in the high-similarity chapter pair; The similarity between the chapter name in the first language and the chapter name in the second language in the high-similarity chapter pair is calculated based on the absolute value of the difference between the chapter number in the chapter name in the first language and the chapter number in the chapter name in the second language and / or the similarity between the chapter title in the chapter name in the first language and the chapter title in the chapter name in the second language.

16. A corpus generation device, characterized in that: include: A first acquisition module is configured to acquire a first language literary work, first language literary work feature data including the title, author, abstract, and chapter name of the first language literary work, and a second language literary work, second language literary work feature data including the title, author, abstract, and chapter name of the second language literary work; a second acquisition module configured to acquire a mapping relationship between the titles of the first language literary works and the titles of the second language literary works, and a mapping relationship between the authors; a first calculation module configured to calculate the similarity between the titles of the first language literary work and the second language literary work, the matching degree between the authors, the similarity between the abstracts, and the similarity between specific parts of the contents; a second calculation module configured to perform weighted calculation on a mapping relationship between the titles of the first language literary work and the second language literary work, a mapping relationship between the authors, similarity between the titles, matching degrees between the authors, similarity between the abstracts, and similarity between specific portions of content, to obtain a literary work pair comprising the first language literary work and the second language literary work; an alignment module configured to align chapters in the first language literary work and the second language literary work in the literary work pair according to chapter contents and chapter names in the first language literary work and the second language literary work; a third acquisition module configured to acquire sentence pairs including corresponding sentences in the first language and sentences in the second language from the aligned passages; A generating module is configured to screen the sentence pairs according to a preset corpus screening rule to generate corpus pairs; Among them, the first calculation module is further configured to: calculate a first similarity score of the similarity between the title of the first language literary work and the title of the second language literary work according to a preset mutual translation dictionary between the first language and the second language; translate the title of the first language literary work according to a preset translation tool to obtain a translation result; perform word segmentation and overlap calculation on the translation result and the title of the second language literary work to be matched, and calculate a second similarity score of the similarity between the title of the first language literary work and the title of the second language literary work; and perform weighted calculation on the first similarity score and the second similarity score to obtain the similarity between the title of the first language literary work and the title of the second language literary work.

17. An electronic device, characterized in that: comprising a memory and a processor; wherein, The memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1-9 and 11-16.

18. A readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1-9 and 11-16 is implemented.

Images