A method, apparatus and medium for extracting web bilingual parallel corpora.

Abstract

This invention belongs to the field of machine translation technology and provides a method, apparatus, and medium for extracting bilingual parallel corpora from the web. The method includes: Step S1: Selecting websites containing parallel corpora from the Internet; Step S2: Obtaining the web page content; Step S3: For each text block, performing sentence segmentation based on language-specific rules; Step S4: Deleting blank and duplicate lines, and identifying the language of each line of text; Step S5: Grouping sentences of the same language according to their original order on the web page to obtain a first language sentence group and a second language sentence group, and calculating the difference A in the number of sentences between the two groups. When a difference in the number of sentences exists, sentence alignment is performed; Step S6: Saving the processed parallel corpus as a parallel corpus library. This invention positions machine translation as an auxiliary alignment tool rather than a corpus generation tool, ensuring the purity of the corpus and its high value as a real-world language example.

Description

Technical Field

[0001] This invention belongs to the field of machine translation technology, specifically, it relates to a method, apparatus and medium for extracting web bilingual parallel corpora. Background Technology

[0002] In the field of machine translation, the quality and richness of parallel corpora have a significant impact on the final effect of machine translation models. Currently, there are many bilingual websites on the Internet. Extracting high-quality parallel corpora directly from these websites faces many technical challenges: (1) Noisy data: Web pages contain non-essential content such as navigation bars, advertisements, copyright notices, and script code, which can seriously pollute the corpus; (2) Structural complexity: Bilingual web pages have diverse layouts and may have problems such as multiple paragraphs to one paragraph, paragraph misalignment, undertranslation (content in one language is missing in another language), and overtranslation (content in one language is over-translated or over-interpreted in another language); (3) Alignment difficulties: Traditional sentence alignment methods (such as the Gale-Church algorithm based on length or word co-occurrence) show a significant decrease in alignment accuracy when dealing with noisy and non-literary web page text.

[0003] In this technical field, some solutions have attempted to use machine translation to directly supplement missing corpora, but this leads to the loss of authenticity of the corpus, introduces biases in machine translation, and fails to meet the demand for native, high-quality parallel corpora. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus and medium for extracting web bilingual parallel corpora, so as to solve the technical problems existing in the prior art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for extracting web bilingual parallel corpora includes the following steps:

[0007] Step S1: Select websites containing parallel corpora from the Internet and save them locally or in a database as a pool of websites to be collected or a file of websites to be collected;

[0008] Step S2: Obtain the web page content, use a web crawler framework to obtain the HTML content of the web page, and then use a parser to extract the text information in the web page. Specifically, paragraphs, lists, and headings in the text information are classified and saved separately, and each paragraph, list item, or heading is saved as an independent text block.

[0009] Step S3: For each text block, sentence segmentation is performed based on language-based sentence segmentation rules. At the same time, a pre-trained sentence boundary detection model is used to assist in sentence segmentation of complex text and to handle quoted or bracketed content.

[0010] Step S4: Delete blank lines and identical duplicate lines from the text; then, for each line of text, use a language detection model to calculate the confidence score of its language, retain sentences with confidence scores higher than a set threshold, and remove sentences with confidence scores lower than the threshold.

[0011] Step S5: Group sentences of the same language according to their original order on the web page to obtain the first language sentence group and the second language sentence group, and calculate the difference A in the number of sentences between the two groups. When there is a difference in the number of sentences, perform sentence alignment.

[0012] Step S6: Save the processed parallel corpus as a parallel corpus library.

[0013] Preferably, in step S5, when there is a difference in the number of sentences, the method for sentence alignment is as follows:

[0014] Step S5.1: Translation assistance: Select the sentence group with the fewest sentences from the first language sentence group and the second language sentence group, and temporarily translate the original sentences of the sentence group into another language through a pre-trained machine translation model to obtain an auxiliary translation sentence set;

[0015] Step S5.2: Similarity matching: For each sentence in the set of auxiliary translation sentences, perform the following steps: Select the sentence group with the most sentences in the first language sentence group and the second language sentence group, and in the original sentences of the sentence group, based on the current position, traverse within a sliding window of size A+1, and use the sentence embedding model to calculate the similarity score between the auxiliary translation sentence and each original sentence in the window;

[0016] Step S5.3: Extraction of real sentence pairs: When there are sentences with a similarity score not lower than the set threshold, retain the pair with the highest score;

[0017] Step S5.4: After the traversal is completed, all successfully matched original sentence pairs are obtained, forming the processed parallel corpus;

[0018] Preferably, the method further includes step S5.5: based on the parallel corpus processed in step S5.4, by calculating the sentence pair length ratio and keyword matching degree, sentence pairs that are over-translated or under-translated are filtered out, wherein sentence pairs whose length ratio exceeds a set range or whose keyword matching degree is lower than a threshold are removed.

[0019] Preferably, in step S5.1, the machine translation model is a Transformer-based neural machine translation model; in step S5.2, the sentence embedding model is generated using a pre-trained multilingual sentence embedding model.

[0020] Preferably, in step S2, the web page content is obtained using scrapy, aiohttp, or requests; the parser is lxml or BeautifulSoup.

[0021] Preferably, in step S3, when splitting sentences, for Chinese text, periods, question marks, and exclamation marks are used as sentence break markers, and for English text, periods, question marks, exclamation marks, and spaces are used as sentence break markers.

[0022] Preferably, in step S3, the sentence boundary detection model is trained based on a bidirectional LSTM model.

[0023] Preferably, in step S4, the language detection model is the langdetect library or the fastText model.

[0024] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the web bilingual parallel corpus extraction method as described above.

[0025] To achieve the above objectives, the present invention also provides an apparatus for extracting web bilingual parallel corpora, comprising: a processor and a memory; the memory for storing a computer program; the processor being connected to the memory and for executing the computer program stored in the memory, so that the apparatus for extracting web bilingual parallel corpora performs the web bilingual parallel corpus extraction method as described above.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) This invention positions machine translation as an auxiliary alignment tool rather than a corpus generation tool. The sentences generated by machine translation are only used as temporary references during the matching process. The final saved sentence pairs are 100% derived from the original web pages, ensuring the purity of the corpus and its high value as a real language example. This is fundamentally different from the existing methods of directly mixing machine translation corpora.

[0028] (2) This invention combines translation assistance and sliding window traversal to predict the possible missing alignment positions, with high alignment accuracy, effectively solving the problems of undertranslation and misalignment.

[0029] (3) This invention preserves the logical structure of the document by classifying and saving paragraphs, lists and headings as independent text blocks, laying the foundation for subsequent accurate alignment, reducing errors of multiple paragraphs to one paragraph. At the same time, combined with rules and sentence boundary detection models, it can effectively handle complex sentences, avoid over-translation or under-translation caused by incorrect sentence segmentation, thoroughly remove noise, and produce high-quality corpus.

[0030] (4) This invention performs secondary filtering by sentence pair length ratio and keyword matching degree, which can eliminate obviously unreasonable alignment results (such as sentence pairs with huge length differences or completely unrelated topics), further improving the quality of the final corpus. Attached Figure Description

[0031] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0032] Figure 2 This is a principle block diagram of Embodiment 3 of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to have a clearer understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described below are merely for illustrative purposes and to facilitate understanding. The technical solutions provided by the present invention are not limited to those provided in the following embodiments, nor should they limit the scope of protection of the present invention.

[0034] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides a method for extracting bilingual parallel corpora from web pages. Its aim is to accurately extract authentic, high-quality bilingual parallel sentence pairs from original web pages even when content is missing or misaligned. Specifically, it includes the following steps:

[0037] I. Selecting Parallel Corpus Webpages

[0038] Select websites containing parallel corpora from the Internet and save them locally or in a database as a pool of websites to be collected or a file of websites to be collected; for example, select a bilingual news website (such as "China Radio International") from the Internet and save its URL list to the local file website_pool.txt as a pool of websites to be collected.

[0039] II. Obtaining web page content and extracting text blocks

[0040] After obtaining the web page content, a crawler is written using the Scrapy framework to read the URLs from website_pool.txt and concurrently download the HTML content of these pages. Then, the lxml parser is used to parse the HTML, and the paragraphs, lists and headings in the text information are saved separately, with each paragraph, list item or heading saved as an independent text block.

[0041] For example, for a news page, the parser will identify and extract... (paragraph), (ListItem), <h1>-< / h1> <h6>Plain text within tags such as (title).

[0042] one This is the first news item in Chinese. It will be saved as a text block: "This is the first Chinese news article".

[0043] one Project 1 Project Two It will be split into two separate text blocks: "Project One" and "Project Two".

[0044] one< / h6> <h2> Chapter 1: Introduction< / h2> will be saved as a text block: "Chapter 1: Introduction”.

[0045] Through the method provided in this step, the logical structure of the document is retained.

[0046] III. Split each text block into sentences

[0047] In this step, sentence splitting is performed based on the language's sentence-breaking rules. The sentence-breaking rules for the language: for Chinese text, use periods, question marks, and exclamation marks as sentence-breaking markers; for English text, use periods, question marks, exclamation marks, and spaces as sentence-breaking markers. For example: for the Chinese text block "您好!请问怎么走?我知道了。”, use periods, question marks, and exclamation marks to split, obtaining three sentences: ["您好!”, "请问怎么走?”, "我知道了。”]; for the English text block "Hello! How are you? I am fine.”, also use the corresponding punctuation and spaces to split.

[0048] At the same time, a pre-trained sentence boundary detection model is used to assist in sentence-breaking for complex texts, handling quoted or bracketed content. The sentence boundary detection model is trained based on a bidirectional LSTM model. For example: for a complex text block "The professor said: ‘This is important.’ Then he left.”, relying solely on the rules may result in incorrect sentence-breaking at the single quotes. At this time, use the pre-trained bidirectional LSTM sentence boundary detection model for analysis. This model is an existing mature model. By learning a large amount of text, it can understand that what follows "said:" is usually a quotation, thus correctly identifying the entire part as a complete sentence and avoiding incorrect splitting.

[0049] IV. Remove noisy data

[0050] In this step, first, delete all blank lines and duplicate lines that are exactly the same;

[0051] Then, use a language detection model to identify the language of each line of text. The language detection model uses existing mature models. Retain the sentences with a confidence level higher than a set threshold (set manually), and remove the sentences with a confidence level lower than this threshold; for example: langdetect language detection model, fastText language detection model, etc., and process according to the confidence level of the language. For example: for the sentence "Welcome to Beijing 欢迎来到北京”, the fastText model calculates that the confidence level for it being English is 0.6, and the confidence level for it being Chinese is 0.4. According to the principle of retaining the language information with a higher confidence level, this sentence will be determined to be English and its original form will be retained (retain the original information of the entire sentence, rather than removing the Chinese part). For strings that are obviously noise or sentences with severe mixing, their highest language confidence level will be very low (such as lower than 0.7), and such sentences will be removed as a whole.

[0052] V. Sentence Alignment

[0053] Group the sentences in the same language according to their original order in the web page to obtain the first language sentence group and the second language sentence group, and calculate the sentence number difference A between the two groups. When there is a sentence number difference (i.e., A > 0), perform sentence alignment. The specific method is as follows:

[0054] Step S5.1: Translation assistance: Temporarily translate the original sentences of the sentence group with the least number of sentences in the first language sentence group and the second language sentence group into another language through a pre-trained machine translation model to obtain an auxiliary translation sentence set. Among them, the machine translation model uses existing mature models, such as the neural machine translation model based on Transformer; the original sentence is the original sentence of the language with fewer sentences;

[0055] Suppose after the above steps, a group containing 5 Chinese original sentences (Zh1, Zh2, Zh3, Zh4, Zh5) and a group containing 4 English original sentences (En1, En2, En3, En4) are obtained, and the sentence number difference A = 1; since the English group has fewer sentences, the 4 English original sentences (En1 - En4) are temporarily translated into Chinese through a neural machine translation model based on Transformer (such as opus-mt-en-zh of Hugging Face) to obtain the auxiliary translation sentence set (T1, T2, T3, T4).

[0056] Step S5.2: Similarity Matching: For each sentence in the auxiliary translation sentence set, in the original sentences of the sentence group with the most sentences in the first language sentence group and the second language sentence group, based on the current position, traverse within a sliding window of size A+1, and use a sentence embedding model to calculate the similarity score between the auxiliary translation sentence and each original sentence in the window. The sentence embedding model adopts an existing mature model, which is generated using a pre-trained multilingual sentence embedding model, such as Sentence-BERT.

[0057] Example: Based on the original order of the English sentences, we iterate through the sentences. For T1 (corresponding to En1), we iterate through the original Chinese sentences (Zh1, Zh2) within a sliding window (size A+1=2). We use a pre-trained multilingual sentence embedding model (such as Sentence-BERT) to calculate the similarity scores between T1 and Zh1, and between T1 and Zh2. We assume that the score between T1 and Zh1 is 0.9, and the score between T1 and Zh2 is 0.3.

[0058] Step S5.3: Extraction of Real Sentence Pairs: When there are sentences with a similarity score not lower than the set threshold, retain the pair with the highest score. For example, if T1 and Zh1 have a score of 0.9, which is higher than the set threshold (e.g., 0.7) and is the highest score, then retain the sentence pair. It should be noted that the original sentence pair (Zh1, En1) is retained, not (T1, En1) or (T1, Zh1). The auxiliary translation sentence T1 is discarded after fulfilling its purpose. Iteration: Then, the same operation is performed on T2 (corresponding to En2), searching for the best match within the sliding window (Zh2, Zh3) of the Chinese sentence. This process is repeated until T4 is processed.

[0059] Step S5.4: After the traversal is complete, obtain all successfully matched original sentence pairs. After the traversal is complete, for example: based on the above example, we obtain all successfully matched original sentence pairs, such as [(Zh1, En1), (Zh3, En2), (Zh4, En3), (Zh5, En4)]. It is worth noting that the Chinese sentence Zh2 was not matched, possibly because it is a transitional sentence or a translation that is actually missing in the English (incomplete translation).

[0060] Step S5.5: Post-processing filtering: By calculating the sentence pair length ratio and keyword matching degree, filter out over-translated or under-translated sentence pairs. Among them, sentence pairs with a length ratio exceeding the set range or a keyword matching degree below the threshold are removed.

[0061] Example: Length comparison: Suppose the length ratio of (Zh5, En4) is abnormal, with Zh5 being very long and En4 being very short (e.g., ratio > 2.5). This may indicate "overtranslation" or alignment error, and this sentence pair will be removed. Keyword matching: High-frequency nouns are extracted as keywords from both Chinese and English sentence groups. For each sentence pair, it is checked whether they share a certain proportion of keywords. If a sentence pair does not have any common keywords, it is considered that its theme does not match and it will be removed.

[0062] Through the steps described above, machine translation is used as an auxiliary alignment tool, rather than the common practice of using machine translation as a corpus generation tool. Based on this approach, potentially missing alignment positions can be inferred, thereby recovering potentially overlooked parallel sentence pairs from the original sentences. Furthermore, the final output corpus originates entirely from the original web pages, ensuring the purity and high value of the corpus. The machine translation model used in this step is only used as an auxiliary reference for calculating similarity during sentence alignment; the translated sentences it generates are temporary and discarded after similarity matching is completed, not constituting part of the final parallel corpus. All parallel sentence pairs output by this method are derived from text extracted from the original web pages, thus guaranteeing the authenticity and high quality of the corpus.

[0063] VI. Preserve parallel corpora

[0064] The final refined parallel sentence pairs (e.g., [(Zh1, En1), (Zh3, En2), (Zh4, En3)]) are saved in TMX (Translation Memory eXchange) or simple TSV (Tab-Separated Values) format to form the final high-quality, highly authentic bilingual parallel corpus.

[0065] Example 2

[0066] This embodiment provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the web bilingual parallel corpus extraction method provided in Embodiment 1. Those skilled in the art will understand that all or part of the steps of the method provided in Embodiment 1 can be implemented by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the method provided in Embodiment 1. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0067] Example 3

[0068] like Figure 2 As shown, this embodiment provides a web bilingual parallel corpus extraction device, including: a processor and a memory; the memory is used to store computer programs; the processor is connected to the memory and is used to execute the computer programs stored in the memory, so that the web bilingual parallel corpus extraction device performs the web bilingual parallel corpus extraction method provided in Embodiment 1.

[0069] Specifically, memory includes various media that can store program code, such as ROM, RAM, magnetic disks, USB flash drives, memory cards, or optical discs.

[0070] Preferably, the processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0071] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for extracting web bilingual parallel corpora, characterized in that, Includes the following steps: Step S1: Select websites containing parallel corpora from the Internet and save them locally or in a database as a pool of websites to be collected or a file of websites to be collected; Step S2: Obtain the web page content, use a web crawler framework to obtain the HTML content of the web page, and then use a parser to extract the text information in the web page. Specifically, paragraphs, lists, and headings in the text information are classified and saved separately, and each paragraph, list item, or heading is saved as an independent text block. Step S3: For each text block, perform sentence splitting based on language segmentation rules. At the same time, use a pre-trained sentence boundary detection model to assist in segmenting complex text and handle quoted or bracketed content. Step S4: Delete blank lines and identical duplicate lines from the text; then, for each line of text, use a language detection model to calculate the confidence score of its language, retain sentences with confidence scores higher than a set threshold, and remove sentences with confidence scores lower than the threshold. Step S5: Group sentences of the same language according to their original order on the web page to obtain the first language sentence group and the second language sentence group, and calculate the difference A in the number of sentences between the two groups. When there is a difference in the number of sentences, perform sentence alignment. Step S6: Save the processed parallel corpus as a parallel corpus database; In step S5, when there is a difference in the number of sentences, the method for sentence alignment is as follows: Step S5.1: Translation assistance: Select the sentence group with the fewest sentences from the first language sentence group and the second language sentence group, and temporarily translate the original sentences of the sentence group into another language through a pre-trained machine translation model to obtain an auxiliary translation sentence set; Step S5.2: Similarity matching: For each sentence in the set of auxiliary translation sentences, perform the following steps: Select the sentence group with the most sentences in the first language sentence group and the second language sentence group, and in the original sentences of the sentence group, based on the current position, traverse within a sliding window of size A+1, and use the sentence embedding model to calculate the similarity score between the auxiliary translation sentence and each original sentence in the window; Step S5.3: Extraction of real sentence pairs: When there are sentences with a similarity score not lower than the set threshold, retain the pair with the highest score; Step S5.4: After the traversal is completed, all successfully matched original sentence pairs are obtained, forming the processed parallel corpus; It also includes step S5.5: Based on the parallel corpus processed in step S5.4, sentence pairs that are over-translated or under-translated are filtered out by calculating the sentence pair length ratio and keyword matching degree. Among them, sentence pairs whose length ratio exceeds the set range or whose keyword matching degree is lower than the threshold are removed.

2. The method for extracting web bilingual parallel corpora according to claim 1, characterized in that, In step S5.1, the machine translation model is a Transformer-based neural machine translation model; in step S5.2, the sentence embedding model is generated using a pre-trained multilingual sentence embedding model.

3. The method for extracting web bilingual parallel corpora according to claim 2, characterized in that, In step S2, the web page content is obtained using scrapy, aiohttp, or requests; the parser is lxml or BeautifulSoup.

4. The method for extracting web bilingual parallel corpora according to claim 3, characterized in that, In step S3, when splitting sentences, for Chinese text, periods, question marks, and exclamation marks are used as sentence break markers, and for English text, periods, question marks, exclamation marks, and spaces are used as sentence break markers.

5. The method for extracting web bilingual parallel corpora according to claim 4, characterized in that, In step S3, the sentence boundary detection model is trained based on a bidirectional LSTM model.

6. The method for extracting web bilingual parallel corpora according to claim 5, characterized in that, In step S4, the language detection model is either the langdetect library or the fastText model.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement the method for extracting web bilingual parallel corpora as described in any one of claims 1 to 6.

8. A device for extracting web bilingual parallel corpora, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the web bilingual parallel corpus extraction apparatus performs the web bilingual parallel corpus extraction method as described in any one of claims 1 to 6.

Images