Parallel corpus construction method and device, equipment and storage medium

Abstract

The application provides a parallel corpus construction method and device, electronic equipment and computer readable storage medium; it relates to artificial intelligence technology; the method comprises the following steps: obtaining a plurality of first sentences of a first text in a first language, and obtaining a plurality of second sentences of a second text in a second language; based on each first sentence, the plurality of second sentences of the second text are aligned and queried, and the second sentence aligned successfully with any first sentence is taken as an aligned sentence; the second sentence that matches the boundary information of the first text and fails to align with the plurality of first sentences is taken as a missed alignment sentence; based on the boundary information of the first text, the aligned sentence and the missed alignment sentence are merged to obtain a third text aligned with the first text; and a bilingual parallel corpus is constructed based on the first text and the third text. Through the application, a complete aligned bilingual parallel corpus can be automatically obtained.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, electronic device, and computer-readable storage medium for constructing parallel corpora. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] Text translation is a crucial application in the field of artificial intelligence. Current solutions involve training a text translation model using a bilingual analogy corpus and then performing text translation based on that model. However, misalignment in the bilingual corpora within these corpora leads to poor training of the text translation model, further resulting in low accuracy in text translation. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and computer-readable storage medium for constructing parallel corpora, which can automatically acquire fully aligned bilingual parallel corpora.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for constructing parallel corpora, including:

[0007] Get multiple first sentences from the first text in the first language, and get multiple second sentences from the second text in the second language;

[0008] Alignment query processing is performed on multiple second sentences of the second text based on each first sentence, and the second sentence that is successfully aligned with any first sentence is taken as the aligned sentence;

[0009] The second sentence that matches the boundary information of the first text and fails to align with all of the first sentences is considered a missing alignment sentence.

[0010] Based on the boundary information of the first text, the aligned sentence and the missing aligned sentence are merged to obtain a third text aligned with the first text;

[0011] A bilingual parallel corpus is constructed based on the first text and the third text.

[0012] The method in the above technical solution further includes:

[0013] The first text is translated using a text translation model to obtain the translated text of the first text;

[0014] Based on the translated text of the first text and the third text, construct the loss function of the text translation model;

[0015] Update the parameters of the text translation model until the loss function converges, and use the updated parameters of the text translation model when the loss function converges as the parameters of the trained text translation model.

[0016] In the above technical solution, the step of performing text translation processing on the first text using the text translation model to obtain the translated text of the first text includes:

[0017] The following processing is performed using the text translation model:

[0018] The first text is encoded to obtain the encoded vector of the first text;

[0019] The encoding vector of the first text is decoded to generate a translation vector corresponding to the first text;

[0020] Based on the order of the generated translation vectors, the translation words corresponding to the translation vectors are combined to obtain the translated text of the first text.

[0021] This application provides a parallel corpus construction apparatus, the apparatus comprising:

[0022] The first acquisition module is used to acquire multiple first sentences of the first text in the first language and multiple second sentences of the second text in the second language;

[0023] The alignment module is used to perform alignment query processing on multiple second sentences of the second text based on each first sentence, and to take the second sentence that is successfully aligned with any of the first sentences as the aligned sentence; and to take the second sentence that matches the boundary information of the first text and fails to align with all of the multiple first sentences as the missing aligned sentence.

[0024] The merging module is used to merge the aligned sentence and the missing aligned sentence based on the boundary information of the first text to obtain a third text aligned with the first text.

[0025] A construction module is used to construct a bilingual parallel corpus based on the first text and the third text.

[0026] In the above technical solution, the alignment module is further used to obtain an alignment list between the first text and the second text;

[0027] The alignment list includes the alignment mapping relationship between sentences in the first text and sentences in the second text;

[0028] Based on any of the first sentences in the first text, query the alignment list and use the sentences in the second text that are found in the query as the alignment sentences.

[0029] In the above technical solution, the alignment module is further used to perform text translation processing on any of the first sentences of the first text through the text translation model to obtain the translated sentence of any of the first sentences;

[0030] The translated sentence is matched with each of the plurality of second sentences, and the second sentence that is successfully matched is taken as the second sentence that is aligned with any of the first sentences;

[0031] Based on a second sentence that aligns with any of the first sentences, construct an alignment list between the first text and the second text.

[0032] In the above technical solution, the alignment module is further used to perform text encoding processing on the translated sentence to obtain the encoding vector of the translated sentence;

[0033] The multiple second sentences are processed by text encoding to obtain the encoding vector of each second sentence;

[0034] The similarity between the encoding vector of the translated sentence and the encoding vector of each second sentence is calculated to obtain the similarity between the translated sentence and each second sentence.

[0035] When any of the aforementioned similarities is greater than the similarity threshold, the second sentence corresponding to any of the aforementioned similarities is taken as the second sentence aligned with any of the first sentences.

[0036] In the above technical solution, the alignment module is further used to perform heuristic alignment processing on the translated sentence and any of the second sentences to obtain a first alignment result of any of the second sentences;

[0037] The translated sentence is length-aligned with any of the second sentences to obtain a second alignment result for any of the second sentences.

[0038] When both the first alignment result and the second alignment result indicate successful alignment, any second sentence is taken as the second sentence aligned with any first sentence.

[0039] In the above technical solution, the alignment module is further used to obtain punctuation marks in the translated sentence and obtain punctuation marks in any of the second sentences;

[0040] The punctuation marks in the translated sentence are compared with the punctuation marks in any of the second sentences, and the comparison result is used as the first alignment result of any of the second sentences.

[0041] In the above technical solution, the alignment module is further used to obtain the sentence length of the translated sentence and the sentence length of any of the second sentences;

[0042] Determine a first ratio between the sentence length of the translated sentence and the sentence length of any of the second sentences;

[0043] The first ratio is matched with a set ratio range, and the matching result is used as the second alignment result for any second sentence.

[0044] In the above technical solution, the device further includes:

[0045] The processing module is used to obtain a list of sentence segments from the first text;

[0046] The sentence segmentation list includes the sentence segmentation order of each of the first sentences;

[0047] Based on the sentence segmentation list, the first sentence and the first sentence in the first text are determined, and the first sentence and the first sentence are used as the boundary information of the first text.

[0048] The second sentence in the second text that corresponds to the first sentence is taken as the second first sentence in the second text, and the second sentence that corresponds to the first last sentence is taken as the second last sentence in the second text;

[0049] The second sentence in the second text, located between the second first sentence and the second last sentence, is taken as the second sentence that matches the boundary information of the first text.

[0050] In the above technical solution, the processing module is further used to perform sentence segmentation processing on the first text based on the set punctuation marks to obtain multiple first sentences of the first text, and record the sentence segmentation order of each first sentence based on the sentence segmentation processing;

[0051] Based on the sentence segmentation order of each of the first sentences, construct a sentence segmentation list for the first text.

[0052] In the above technical solution, the merging module is further used to take the aligned sentences that match the boundary information of the first text as the sentences to be merged;

[0053] According to the order of the sentences to be merged and the missing alignment sentences, the sentences to be merged and the missing alignment sentences are merged to obtain a third text that is aligned with the first text.

[0054] In the above technical solution, the first text includes a first paragraph and a second paragraph adjacent to the first paragraph. The first paragraph is a first sentence, and the second paragraph includes at least one first sentence. The alignment module is further configured to merge the first paragraph and the second paragraph to obtain a pseudo paragraph of the first text when the first paragraph fails to align with the plurality of second sentences. The boundary information of the pseudo paragraph is the boundary information of the first text.

[0055] The construction module is also used to construct a bilingual parallel corpus based on the pseudo-paragraph and the third text.

[0056] In the above technical solution, the device further includes:

[0057] The training module is used to perform text translation processing on the first text using a text translation model to obtain the translated text of the first text;

[0058] Based on the translated text of the first text and the third text, construct the loss function of the text translation model;

[0059] Update the parameters of the text translation model until the loss function converges, and use the updated parameters of the text translation model when the loss function converges as the parameters of the trained text translation model.

[0060] In the above technical solution, the training module is also used to perform the following processing through the text translation model:

[0061] The first text is encoded to obtain the encoded vector of the first text;

[0062] The encoding vector of the first text is decoded to generate a translation vector corresponding to the first text;

[0063] Based on the order of the generated translation vectors, the translation words corresponding to the translation vectors are combined to obtain the translated text of the first text.

[0064] This application provides an electronic device for constructing parallel corpora, the electronic device comprising:

[0065] Memory, used to store executable instructions;

[0066] The processor, when executing executable instructions stored in the memory, implements the parallel corpus construction method provided in the embodiments of this application.

[0067] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the parallel corpus construction method provided in this application.

[0068] The embodiments of this application have the following beneficial effects:

[0069] Based on the boundary information of the first text in the first language, missing alignment sentences in the second text are identified. Based on the boundary information of the first text, the aligned sentences that are successfully aligned with the first text are merged with the missing alignment sentences, thereby automatically obtaining the third text aligned with the first text. This ensures the integrity of the third text, improves the accuracy of text alignment, and provides more accurate bilingual parallel corpora to improve the training effect of the text translation model, which is convenient for subsequent improvement of the accuracy of text translation. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of a web page provided in an embodiment of this application;

[0071] Figure 2A It is a Chinese-language analogy corpus provided by related technologies;

[0072] Figure 2B It is an English-language analogy corpus provided by related technologies;

[0073] Figure 3 This is a schematic diagram illustrating an application scenario of the text translation system provided in this application embodiment;

[0074] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0075] Figures 5A-5B This is a flowchart illustrating the parallel corpus construction method provided in the embodiments of this application;

[0076] Figure 6 This is a schematic diagram of the alignment list provided in an embodiment of this application;

[0077] Figure 7 This is a schematic diagram of the structure of the text translation model provided in the embodiments of this application;

[0078] Figure 8 This is a flowchart illustrating the alignment method provided in an embodiment of this application;

[0079] Figure 9 This is a schematic diagram of pseudo-paragraph missing alignment recall provided in the embodiments of this application;

[0080] Figure 10 This is a schematic diagram of the logic flow of the alignment method provided in the embodiments of this application;

[0081] Figure 11 This is a schematic diagram illustrating the construction process of a bilingual corpus of Chinese and English novel passages provided in an embodiment of this application;

[0082] Figure 12 This is a flowchart of the paragraph information extraction module provided in an embodiment of this application;

[0083] Figure 13 This is a logic flowchart of the sentence alignment module provided in the embodiments of this application. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0085] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0086] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0087] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0088] 1) Text Translation Model: A model built based on artificial intelligence principles for text translation processing. The type of text translation model is not limited; for example, it can be an artificial neural network model. In the embodiments of this application, the text translation model can be a deep neural network (DNN), a recurrent neural network (RNN), a self-attention neural network (SAN), etc.

[0089] 2) Parallel corpora: A bilingual or multilingual corpus consisting of source language texts and their parallel corresponding translation texts. The alignment level can be word-level, sentence-level, paragraph-level, or text-level.

[0090] 3) Comparable corpora: Two or more corpora consisting of texts in different languages ​​or different variants of the same language. Comparable corpora can be further subdivided into monolingual, bilingual, and multilingual corpora. Monolingual corpora collect texts with similar content in similar environments within the same language, such as the English Comparable Corpus (ECC) constructed by Loviosa. Bilingual and multilingual corpora collect texts in different languages ​​that are similar in content, register, and communicative context, and are often used in contrastive linguistics.

[0091] 4) Loss value: Used to represent the difference between the model's output and the actual result (or labeled result). Model training mainly involves two processes: forward propagation and back propagation. Taking a neural network model including an input layer, hidden layer, and output layer as an example, forward propagation refers to processing the data sequentially through the input layer, hidden layer, and output layer to obtain the final output result; back propagation refers to propagating the calculated loss value sequentially to the output layer, hidden layer, and input layer, thereby updating the weight parameters in each layer.

[0092] 5) Self-training: Through a series of steps, the existing labeled data is used to label the remaining unlabeled data, thereby increasing the amount of training data.

[0093] Online novels refer to written literary works published and disseminated via the internet. Compared to traditional novels, online novels use more colloquial language and are filled with internet slang. One of the most common types is the serialized novel, where the author updates the plot of a novel online periodically. Over the past decade, Chinese online literature has begun to go global through licensing agreements, such as... Figure 1 The images show various online novels exported overseas. China has over 24 million online novels in its inventory, with more than 17 million online writers, but fewer than 300 have actually been exported. The cost of translating a single online novel is as high as 600,000 to 800,000 yuan. Therefore, the translation of online novels urgently needs to leverage artificial intelligence technology to improve translation efficiency and reduce costs.

[0094] On the other hand, deep neural networks have become a core model in the field of artificial intelligence, widely used in speech recognition, computer vision, natural language processing, and other areas. Neural network models have demonstrated outstanding performance in various tasks, particularly machine translation. In the field of machine translation, neural machine translation models (NMT) based on the encoder-decoder framework have seen rapid development in recent years, with performance far exceeding that of statistical machine translation models (SMT). This has brought the possibility of automatically translating highly challenging literary works.

[0095] As is well known, deep learning is a standard data-driven approach that treats deep networks as a black box, relying on massive amounts of data to solve real-world problems. Therefore, the main factor determining the performance of machine translation systems for online novels is the size of the relevant data. Due to copyright and other reasons, there are currently no publicly available, directly accessible parallel corpora of online novels. However, because online novels are currently very popular overseas, many are translated by human translators and published on the internet, resulting in a considerable amount of analogical corpora on the internet. For example... Figure 2A-2B As shown, the Chinese version of the same novel (such as...) Figure 5A (as shown) and English version (as shown) Figure 5B The texts shown can all be found on the internet, but they are not parallel corpora. Therefore, they cannot be directly used to train machine translation models. They need to be aligned with sentences in the Chinese and English novels using an alignment tool before they can be directly used to train machine translation models (text translation models).

[0096] In related technologies, the following problems may occur when aligning sentences in original novel chapters:

[0097] 1) Sentence segmentation usually results in incomplete sentences, and sentence segmentation preprocessing leads to incomplete sentences in partially aligned corpora;

[0098] 2) Due to biases in the translation field, the translation of source sentences is inaccurate, and the final alignment algorithm bias leads to misalignment and missing alignment problems in the corpus.

[0099] To address the aforementioned issues, embodiments of this application provide a method for constructing bilingual corpora, a text translation method, an apparatus, an electronic device, and a computer-readable storage medium, which can automatically acquire fully aligned bilingual corpora and improve the accuracy of text translation.

[0100] The parallel corpus construction method provided in this application embodiment can be implemented by the terminal / server alone; or it can be implemented by the terminal and the server in collaboration. For example, the terminal can undertake the parallel corpus construction method described below alone, or the terminal can send a training request for the text translation model to the server, and the server can execute the parallel corpus construction method according to the received training request for the text translation model, and perform text translation tasks based on the trained text translation model.

[0101] The electronic device for parallel corpus construction provided in this application can be various types of terminal devices or servers. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0102] Taking servers as an example, such as server clusters deployed in the cloud, AI as a Service (AIaaS) is provided to users. The AIaaS platform breaks down several common AI services and provides them as independent or packaged services in the cloud. This service model is similar to an AI-themed marketplace. All users can access and use one or more artificial intelligence services provided by the AIaaS platform through application programming interfaces.

[0103] For example, one type of AI cloud service could be a parallel corpus construction service, where a cloud server encapsulates the parallel corpus construction program provided in this application embodiment. Users invoke the parallel corpus construction service in the cloud service through a terminal (running a client, such as a novel client, search client, or in-vehicle client), causing the cloud-deployed server to call the encapsulated parallel corpus construction program. Based on the boundary information of the first text, aligned and misaligned sentences are merged to obtain a third text aligned with the first text. A text translation model is trained based on the bilingual parallel corpus consisting of the first text in the first language and the third text in the second language. The trained text translation model is then used to perform text translation, achieving accurate text translation for subsequent novel applications, in-vehicle applications, etc.

[0104] See Figure 3 , Figure 3This is a schematic diagram of the application scenario of the text translation system 10 provided in the embodiments of this application. The terminal 200 is connected to the server 100 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0105] Terminal 200 (running a client, such as a novel client) can be used to obtain the text to be translated in the first language. For example, the user inputs the text to be translated in the first language through the terminal, and the terminal automatically obtains the text translation request for the text to be translated.

[0106] In some embodiments, a parallel corpus construction plugin can be embedded in the client running on the terminal to implement the parallel corpus construction method locally on the client. For example, the terminal 200 calls the parallel corpus construction plugin to implement the parallel corpus construction method. Based on the boundary information of the first text, aligned sentences and missing aligned sentences are merged to obtain a third text aligned with the first text. A text translation model is trained based on the bilingual parallel corpus consisting of the first text in the first language and the third text in the second language. Based on the text translation request for the text to be translated, the trained text translation model is called, and the text is translated using the trained text translation model to obtain the translated text of the text to be translated, in response to the text translation request for the text to be translated, to achieve accurate text translation for subsequent novel applications, in-vehicle applications, etc.

[0107] As an example, for a novel application, a user inputs an English novel to be translated through a terminal. The trained text translation model then translates the text to obtain a translated Chinese version of the English novel. This responds to text translation requests for English novels, achieving accurate text translation so that users can enjoy a smooth reading experience based on the Chinese version, thus improving the user experience.

[0108] It is worth noting that the trained text translation model can be stored locally on terminal 200 and retrieved when needed. Furthermore, the text to be translated can be either pre-stored locally on terminal 200 or retrieved in real time by terminal 20.

[0109] In some embodiments, after the terminal 200 receives a text translation request for the text to be translated, it calls the parallel corpus construction interface of the server 100 (which can be provided as a cloud service, i.e., a text translation model training service). Based on the boundary information of the first text, the server 100 merges aligned sentences and missing aligned sentences to obtain a third text aligned with the first text. The server trains a text translation model based on the bilingual parallel corpus consisting of the first text in the first language and the third text in the second language. Based on the text translation request for the text to be translated, the server calls the trained text translation model and performs text translation through the trained text translation model to obtain the translated text of the text to be translated, thereby responding to the text translation request for the text to be translated and achieving accurate text translation for subsequent novel applications, in-vehicle applications, etc.

[0110] As an example, in a search application, when a user searches for information that is not in their native language, the search results need to be translated. For instance, if a user finds English information on the search application, the application automatically recognizes the English information and displays a prompt, such as "Do you need it translated into Chinese?" When the user confirms the translation, the application automatically retrieves the text translation request for the English information and sends it to server 100. Server 100 parses the request, obtains the English information, and translates it using a trained text translation model to produce the translated text (Chinese information). This translated text is then sent to terminal 20, where it is displayed on the terminal's screen. This process responds to the English information translation request, achieving accurate text translation so that users can read the information fluently in Chinese, thus improving the user experience.

[0111] It is worth noting that the trained text translation model can be stored locally on server 100 and retrieved when needed.

[0112] In some embodiments, the terminal or server can implement the parallel corpus construction method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0113] This application's embodiments can be implemented using cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form a resource pool, available on demand, offering flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of the technology network system require substantial computing and storage resources.

[0114] In some embodiments, multiple servers can form a blockchain, and server 100 is a node on the blockchain. Information connections can exist between each node in the blockchain, and information can be transmitted between nodes through these connections. The data related to the parallel corpus construction method provided in this application embodiment (e.g., the logic of parallel corpus construction, the training logic of the text translation model, translated text, etc.) can be stored on the blockchain. Because the blockchain has the characteristic of being immutable, the accuracy of the data in the blockchain can be guaranteed. Electronic devices can send query requests to the blockchain to query the data stored in the blockchain.

[0115] The structure of the electronic device provided in the embodiments of this application is described below. See also... Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device 500 for parallel corpus construction provided in an embodiment of this application. The example given is that the electronic device 500 is a server. Figure 4 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general labeled all buses as Bus System 540.

[0116] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0117] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0118] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0119] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0120] The network communication module 553 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0121] In some embodiments, the parallel corpus construction apparatus provided in this application can be implemented in software. Figure 4 A parallel corpus construction apparatus 555 stored in memory 550 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a first acquisition module 5551, an alignment module 5552, a merging module 5553, a construction module 5554, a training module 5555, and a processing module 5556. These modules are logically linked and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0122] In some embodiments, the text translation device provided in this application can be implemented in software, that is, the text translation device stored in the memory 550 can be software in the form of programs and plug-ins, including the following software modules: a second acquisition module and a translation module. These modules are logically linked, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below. It is worth noting that, apart from the text translation device shown, the other structures in the electronic device for text translation can be the same as those in the application. Figure 4 same.

[0123] It should be noted that the parallel corpus construction device 555 and the text translation device can be integrated on one electronic device, that is, the electronic device can simultaneously implement the parallel corpus construction method and the text translation method; or the parallel corpus construction device 555 and the text translation device can be integrated on two separate electronic devices, that is, the electronic device implements either the parallel corpus construction method or the text translation method.

[0124] As mentioned above, the parallel corpus construction method provided in this application can be implemented by various types of electronic devices. See also Figure 5A , Figure 5A This is a flowchart illustrating the parallel corpus construction method provided in the embodiments of this application, combined with... Figure 5A The steps shown are explained.

[0125] In the following steps, the first text (including at least one paragraph) is in a first language, and the second text is in a second language. The first language and the second language are different, for example, the first language is the source language (e.g., Chinese) and the second language is the translation language (e.g., English), or the first language is the translation language (e.g., English) and the second language is the source language (e.g., Chinese).

[0126] In step 101, multiple first sentences of the first text in the first language are obtained, and multiple second sentences of the second text in the second language are obtained.

[0127] As an example of obtaining the first and second texts, when a user needs to train a text translation model (i.e., a translation model) to perform text translation using the trained text translation model, the user inputs the address of the bilingual corpus (the first text in the first language and the second text in the second language) through the terminal. The terminal automatically obtains the training request for the text translation model (including the address of the bilingual corpus) and sends the training request for the text translation model. The server parses the training request for the text translation model and obtains the first text in the first language and the second text in the second language.

[0128] For example, after obtaining the first text in the first language and the second text in the second language, the sentence segmentation tool is used to segment the first text in the first language and the second text in the second language to obtain multiple first sentences in the first text and multiple second sentences in the second text. The multiple first sentences in the first text and multiple second sentences in the second text are saved for subsequent alignment query operations.

[0129] It should be noted that the bilingual corpus (the first text in the first language and the second text in the second language) can be pre-stored locally on the terminal or server, or it can be obtained in real time through web crawler software.

[0130] In step 102, an alignment query is performed on multiple second sentences of the second text based on each first sentence, and the second sentence that is successfully aligned with any first sentence is taken as the aligned sentence.

[0131] For example, if an alignment query operation is performed on multiple second sentences of the second text for any first sentence, and a second sentence that is aligned with any first sentence is found, then the second sentence aligned with any first sentence is taken as the aligned sentence of the first sentence.

[0132] See Figure 5B , Figure 5B This is an optional flowchart illustrating the parallel corpus construction method provided in this embodiment of the invention. Figure 5B Show Figure 5A Step 102 in the process can be achieved through Figure 5B Steps 1021 to 1022 shown are implemented as follows: In step 1021, an alignment list of the first text and the second text is obtained; wherein, the alignment list includes the alignment mapping relationship between sentences in the first text and sentences in the second text; In step 1022, the alignment list is queried based on any first sentence in the first text, and the queried sentence in the second text is used as the aligned sentence.

[0133] like Figure 6 As shown, the alignment tool automatically aligns sentences in the first text with sentences in the second text, thus establishing an alignment mapping relationship between sentences in the first text and sentences in the second text. For example, the alignment mapping relationship includes: {Chinese clause 1 aligns with English clause 1 and English clause 2, Chinese clause 2 aligns with English clause 4 and English clause 5, ..., Chinese clause n aligns with English clause 2n}. Then, based on the first sentence in the first text, the alignment list is queried, and the sentence in the second text found is used as the aligned sentence of the first sentence.

[0134] In some embodiments, obtaining an alignment list between the first text and the second text includes: performing text translation processing on any first sentence of the first text using a text translation model to obtain a translated sentence of any first sentence; matching the translated sentence with multiple second sentences respectively, and taking the successfully matched second sentence as the second sentence aligned with any first sentence; and constructing an alignment list between the first text and the second text based on the second sentences aligned with any first sentence.

[0135] For example, the first sentence in the first text is translated using an initial text translation model to obtain a translated sentence in the second language. The translated sentence is then matched with multiple second sentences in the second text. When a match is successful, it means that the matched second sentence is aligned with the first sentence. All second sentences aligned with the first sentence are recorded and stored in a list to obtain an alignment list between the first and second texts.

[0136] In some embodiments, matching the translated sentence with multiple second sentences respectively, and taking the successfully matched second sentence as the second sentence aligned with any first sentence, includes: performing text encoding processing on the translated sentence to obtain the encoding vector of the translated sentence; performing text encoding processing on the multiple second sentences respectively to obtain the encoding vector of each second sentence; performing similarity processing on the encoding vector of the translated sentence and the encoding vector of each second sentence to obtain the similarity between the translated sentence and each second sentence respectively; when any similarity is greater than a similarity threshold, taking the second sentence corresponding to any similarity as the second sentence aligned with any first sentence.

[0137] The similarity calculation can be a standard method for machine translation evaluation (BLEU, Bilingual Evaluation Understudy), cosine similarity, etc.

[0138] In some embodiments, the translated sentence is matched with a plurality of second sentences respectively, and the successfully matched second sentence is taken as the second sentence aligned with any first sentence, including: performing heuristic alignment processing on the translated sentence and any second sentence to obtain a first alignment result of any second sentence; performing length alignment processing on the translated sentence and any second sentence to obtain a second alignment result of any second sentence; when both the first alignment result and the second alignment result indicate successful alignment, the second sentence is taken as the second sentence aligned with any first sentence.

[0139] Following the example above, the heuristic alignment process is as follows: obtain the punctuation marks in the translated sentence and obtain the punctuation marks in any second sentence; compare the punctuation marks in the translated sentence with the punctuation marks in any second sentence, and use the comparison result as the first alignment result of any second sentence.

[0140] For example, heuristic alignment refers to determining whether sentences are aligned using manually defined matching rules. If all punctuation marks in the translated sentence are present in any second sentence, then the translated sentence corresponds to any second sentence, meaning the first sentence and any second sentence are successfully aligned. For instance, if the translated sentence contains both "?" and "..." symbols, and the second sentence i contains "?", ",", and "..." symbols, then the translated sentence corresponds to the second sentence i, meaning the first sentence and the second sentence i are successfully aligned.

[0141] Following the example above, the length alignment process is as follows: obtain the sentence length of the translated sentence and the sentence length of any second sentence; determine a first ratio between the sentence length of the translated sentence and the sentence length of any second sentence; perform matching processing between the first ratio and a set ratio range, and use the matching result as the second alignment result of any second sentence.

[0142] For example, length alignment refers to the use of word ratios to determine whether sentences are aligned. For instance, if the set ratio range is [0.8, 1.2], then if the first ratio of the length of the translated sentence to the length of any second sentence falls within the set ratio range, then the match is successful, meaning the second alignment result of any second sentence is a successful match; if the first ratio of the length of the translated sentence to the length of any second sentence does not fall within the set ratio range, then the match fails, meaning the second alignment result of any second sentence is a failed match.

[0143] In step 103, the second sentence that matches the boundary information of the first text and fails to align with multiple first sentences is identified as the missing alignment sentence.

[0144] For example, if an alignment query operation is performed on multiple second sentences of the second text for any first sentence, and no second sentence is found that is aligned with any first sentence, and any second sentence matches the boundary information of the first text, then any second sentence is considered a missing alignment sentence.

[0145] In some embodiments, before considering a second sentence that matches the boundary information of the first text but fails to align with multiple first sentences as a missing alignment sentence, a sentence segmentation list of the first text is obtained; wherein, the sentence segmentation list includes the sentence segmentation order of each first sentence; based on the sentence segmentation list, the first first sentence and the first last sentence in the first text are determined, and the first first sentence and the first last sentence are used as the boundary information of the first text; the second sentence in the second text that corresponds to the first first sentence is used as the second first sentence in the second text, and the second sentence that corresponds to the first last sentence is used as the second last sentence in the second text; the second sentence in the second text that is between the second first sentence and the second last sentence is used as the second sentence that matches the boundary information of the first text.

[0146] For example, when the first text consists of multiple paragraphs, the boundary information of the first text is the boundary information of each paragraph (i.e., the first and last sentences of each paragraph). The sentence list of the first text is {Paragraph 1 (Chinese sentence 1, Chinese sentence 2), Paragraph 2 (Chinese sentence 3, Chinese sentence 4, Chinese sentence 5)}. Then, Chinese sentence 1 is taken as the first sentence of paragraph 1, Chinese sentence 2 as the last sentence of paragraph 1, Chinese sentence 3 as the first sentence of paragraph 2, and Chinese sentence 5 as the last sentence of paragraph 2. Thus, Chinese sentences 1 and 2 are the boundary information of paragraph 1 in the first text, and Chinese sentences 3 and 5 are the boundary information of paragraph 2 in the first text. When the second text is {English sentence 1, English sentence 2, English sentence 3}, when English sentence 1 corresponds to the first sentence of paragraph 2 and English sentence 3 corresponds to the last sentence of paragraph 2, then English sentences 1, 2, and 3 are taken as the second sentence that matches the boundary information of paragraph 2.

[0147] Following the example above, obtaining the sentence segmentation list of the first text includes: performing sentence segmentation processing on the first text based on the set punctuation marks to obtain multiple first sentences of the first text, and recording the sentence segmentation order of each first sentence based on the sentence segmentation processing; and constructing the sentence segmentation list of the first text based on the sentence segmentation order of each first sentence.

[0148] For example, by using a sentence segmentation tool in conjunction with set punctuation marks (such as "!", ".", "?", etc.), the first text is segmented into multiple first sentences. The process of segmentation is recorded, thus recording the segmentation order of each first sentence. The segmentation order of all first sentences is then saved in a list, resulting in a sentence segmentation list of the first text. For example, the sentence segmentation list of the first text could be {Paragraph 1 (Chinese sentence 1, Chinese sentence 2), Paragraph 2 (Chinese sentence 3, Chinese sentence 4, Chinese sentence 5)}, meaning the segmentation order of the Chinese sentences is (Chinese sentence 1 → Chinese sentence 2 → Chinese sentence 3 → Chinese sentence 4 → Chinese sentence 5).

[0149] In some embodiments, the first text includes a first paragraph and a second paragraph adjacent to the first paragraph. The first paragraph is a first sentence, and the second paragraph includes at least one first sentence. After performing alignment query processing on multiple second sentences of the second text based on each first sentence, when the first paragraph fails to align with multiple second sentences, the first paragraph and the second paragraph are merged to obtain a pseudo paragraph of the first text. The boundary information of the pseudo paragraph is the boundary information of the first text.

[0150] For example, since some sentences in the text are in separate paragraphs, the paragraph boundary information cannot recall the missing sentences in cases of misalignment, which is a legacy problem of misalignment. Based on this, the embodiments of this application further refine the handling of misalignment. When the first paragraph of the first text is empty (i.e., no second sentence aligned with the first paragraph is found), a precise misalignment repair scheme is needed. This involves locating the empty sentence position (the first paragraph), merging the first paragraph with the adjacent second paragraph to construct a pseudo-paragraph of the first text, using the first and last sentences of the pseudo-paragraph as the boundary information of the first text, and determining the missing aligned sentences in the second text based on the first and last sentences of the pseudo-paragraph. Then, based on the first and last sentences of the pseudo-paragraph, the aligned sentences and the missing aligned sentences are merged to obtain a third text aligned with the pseudo-paragraph. Finally, the missing aligned sentences in the second text are recalled to this pseudo-paragraph, thus constructing a fully aligned bilingual parallel corpus based on the pseudo-paragraph and the third text.

[0151] like Figure 9 As shown, Chinese clause i represents the first paragraph, and Chinese clauses i-1 and i+1 represent the second paragraph. Based on the automatic alignment algorithm, the first English sentence corresponding to Chinese clause i-1 is aligned to clause 1. i-1 The Chinese clause i+1 corresponds to the English last clause aligned with clause n. i+1 Since the Chinese clause i is empty, we merge the Chinese clauses i-1, i, and i+1 to obtain a pseudo-paragraph in Chinese. This allows us to recall the misaligned Chinese clause i and align the first sentence of the English text with clause 1. i-1 1. Align the English last sentence with the clause n i+1 By merging the sentences in the Chinese and English pseudo-parallel texts, a bilingual parallel corpus is constructed based on the Chinese and English pseudo-parallel texts.

[0152] As an example, some paragraphs in the first text may have missing alignment in their first or last sentence, leading to missing alignment in the final merged third text. The paragraph boundary information cannot recall the missing sentences, a legacy issue of missing alignment. Based on this, this application embodiment further refines the handling of missing alignment. For example, if the first text includes a third paragraph and a fourth paragraph adjacent to it, when the first or last sentence of the third paragraph fails to align with multiple second sentences (i.e., is empty), the third and fourth paragraphs are merged to construct a pseudo-paragraph of the first text. The first and last sentences of the pseudo-paragraph are used as boundary information for the first text. Based on the first and last sentences of the pseudo-paragraph, missing aligned sentences in the second text are determined. Then, based on the first and last sentences of the pseudo-paragraph, aligned sentences and missing aligned sentences are merged to obtain a third text aligned with the pseudo-paragraph. The missing aligned sentences in the second text are then recalled to this pseudo-paragraph, thus constructing a fully aligned bilingual parallel corpus based on the pseudo-paragraph and the third text.

[0153] In step 104, based on the boundary information of the first text, the aligned sentences and the missing aligned sentences are merged to obtain the third text aligned with the first text.

[0154] For example, after identifying missing alignment sentences in the second text, based on the boundary information of the first text, the aligned sentences in the second text corresponding to the boundary information of the first text are identified, and the missing alignment sentences are merged with the aligned sentences corresponding to the boundary information of the first text to obtain a complete paragraph aligned with the first text. This allows for the subsequent construction of bilingual parallel corpora based on the accurate third text and the first text, and the training of the text translation model to improve the training effect of the text translation model.

[0155] In some embodiments, based on the boundary information of the first text, the aligned sentences and the missing aligned sentences are merged to obtain a third text aligned with the first text, including: taking the aligned sentences that match the boundary information of the first text as sentences to be merged; and merging the sentences to be merged and the missing aligned sentences in the order of the sentences to be merged and the missing aligned sentences to obtain a third text aligned with the first text.

[0156] For example, the boundary information of the first text is Chinese sentence 1 and Chinese sentence 2. The second sentence in the second text is {English sentence 1, English sentence 2, English sentence 3, English sentence 4, English sentence 5, English sentence 6}. Among them, {English sentence 1, English sentence 2, English sentence 4, English sentence 5} are aligned sentences, and English sentence 3 is a missing aligned sentence. English sentence 2 is aligned with Chinese sentence 1, and English sentence 5 is aligned with Chinese sentence 3. Therefore, English sentence 2, English sentence 4, and English sentence 5 are aligned sentences (sentences to be merged) that match the boundary information of the first text. Based on the order of the sentences to be merged and the missing aligned sentences, English sentence 2, English sentence 3, English sentence 4, and English sentence 5 are merged to obtain a third text that is fully aligned with the first text.

[0157] In step 105, a text translation model is trained based on the first text and the third text.

[0158] For example, bilingual parallel corpora can be constructed based on the first text and a fully aligned third text to train a text translation model. In this way, the training effect of the text translation model can be improved by using fully aligned bilingual corpora, thereby improving the accuracy of text translation.

[0159] In some embodiments, training a text translation model based on a first text and a third text includes: performing text translation processing on the first text using the text translation model to obtain the translated text of the first text; constructing a loss function for the text translation model based on the translated text of the first text and the third text; updating the parameters of the text translation model until the loss function converges, and using the updated parameters of the text translation model when the loss function converges as the parameters of the trained text translation model.

[0160] For example, a loss function for a text translation model can be constructed based on the translated text of the first text and the third text. Where y represents the third text. The first text represents the translated text. The parameters of the text translation model are updated until the loss function converges. The parameters obtained when the loss function converges are used as the parameters of the trained text translation model.

[0161] Specifically, after determining the value of the loss function of the text translation model based on the translated text of the first text and the third text, it can be determined whether the value of the loss function exceeds a preset threshold. When the value of the loss function exceeds the preset threshold, the error signal of the text translation model is determined based on the loss function, the error information is backpropagated in the text translation model, and the model parameters of each layer are updated during the propagation process.

[0162] Here, we explain backpropagation. Training sample data is input into the input layer of a neural network model, passes through the hidden layers, and finally reaches the output layer to output the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated and backpropagated from the output layer to the hidden layers until it reaches the input layer. During the backpropagation process, the values ​​of the model parameters are adjusted according to the error. That is, a loss function is constructed based on the error between the output result and the actual value, and the partial derivatives of the loss function with respect to the model parameters are calculated layer by layer to generate the gradient of the loss function with respect to the model parameters of each layer. Since the direction of the gradient indicates the direction of error amplification, the gradient of the model parameters is inverted and summed with the original parameters of each layer. The summation result is used as the updated model parameters of each layer, thereby reducing the error caused by the model parameters. The above process is iterated until convergence. The text translation model is a neural network model.

[0163] In some embodiments, the text translation process of the first text using a text translation model to obtain the translated text of the first text includes: performing the following processing using the text translation model: encoding the first text to obtain an encoding vector of the first text; decoding the encoding vector of the first text to generate a translation vector corresponding to the first text; and combining the translation words corresponding to the translation vectors based on the order of the generated translation vectors to obtain the translated text of the first text.

[0164] For example, a text translation model can be a recurrent neural network (RNN), which transforms sequence modeling into temporal modeling by circulating the state within its own network. It can also be a sequence-to-sequence generation model structure such as a self-attention network (SAN).

[0165] For example, given the first text x = x1, ..., x I Below, directly optimize its target sentence (translated text) y = y1, ..., y J The conditional probability of is calculated using the formula shown in formula (1):

[0166]

[0167] Where θ are the parameters of the model, y ＜j Let P(y|x) represent the partially generated translated text, and let P(y|x) represent the framework of the text translation model. The training process of the model's parameters is to maximize the training samples D. The probability of is calculated using the formula shown in formula (2):

[0168]

[0169] like Figure 7 As shown, the process of generating the translation vector corresponding to the first text is as follows: the encoded vector of the first text (encoded by the encoder) is decoded to generate the first translation vector corresponding to the first text; the encoded features of the first word corresponding to the first translation vector and the encoded vector of the first text are decoded by the decoder to generate the i-th translation vector corresponding to the first text; the encoded features of the i-th translation vector and the encoded vector of the first text are decoded by the decoder to generate the (i+1)-th encoded vector corresponding to the first text; where 2≤i≤N-1, i is a natural number, and N is the number of translation vectors corresponding to the first text.

[0170] like Figure 7 As shown, the encoder of the text translation model obtains the encoding vector of the first text, where the first text includes M words, and the encoding vector includes the word vector of each word, where M is an integer greater than or equal to 1. Based on the encoding vector, the attention network of the text translation model obtains the context of the first text for the t-th word, where t is an integer greater than or equal to 1 and less than or equal to M. Based on the context of the first text for the t-th word, the decoder state of the t-th word is obtained through the decoder of the text translation model. Based on the decoder state of the t-th word and the encoding vector, the output layer of the text translation model obtains the translation vector corresponding to the t-th word. After obtaining the translation vectors corresponding to the M words, the translation words corresponding to the translation vectors are combined based on the order of the generated translation vectors to obtain the translated text of the first text.

[0171] As mentioned above, the text translation method provided in this application embodiment can be implemented by various types of electronic devices. The text translation method provided in this application embodiment is implemented through the following steps:

[0172] It should be noted that after training the text translation model using the above-mentioned parallel corpus construction method, the trained text translation model is used for subsequent translation tasks.

[0173] In step 201, the text to be translated in the first language is obtained.

[0174] As an example of obtaining text to be translated, a user can input the text to be translated in the first language through the terminal. The terminal automatically obtains the text translation request for the text to be translated. The terminal can call the trained text translation model locally to perform text translation, or the terminal can send the text translation request for the text to be translated to the server, and the server calls the trained text translation model to perform text translation.

[0175] In step 202, the text to be translated is processed by a text translation model to obtain the translated text.

[0176] For example, a text translation model is a sequence-to-sequence generation model structure. The trained text translation model encodes the text to be translated to obtain the encoded vector of the text to be translated. The encoded vector of the text to be translated is then decoded to generate the translation vector corresponding to the text to be translated. Based on the order of the generated translation vectors, the translation words corresponding to the translation vectors are combined to obtain the translated text of the text to be translated.

[0177] It should be noted that the process of generating translation vectors corresponding to the text to be translated is as follows: The encoded vector of the text to be translated (encoded by an encoder) is decoded to generate the first translation vector corresponding to the text to be translated. The encoded features of the first word corresponding to the first translation vector and the encoded vector of the text to be translated are decoded by a decoder to generate the i-th translation vector corresponding to the text to be translated. The encoded features of the i-th translation vector and the encoded vector of the text to be translated are decoded by a decoder to generate the (i+1)-th encoded vector corresponding to the text to be translated. Wherein, 2≤i≤N-1, i is a natural number, and N is the number of translation vectors corresponding to the text to be translated.

[0178] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0179] Chinese online literature has become a vital force in helping China become a world-leading publishing power. The national initiative to "promote Chinese culture abroad" has further boosted the global market for Chinese online literature, guiding it to fully leverage its high-quality content and play a more significant role in the global cultural and creative industries. In recent years, the export of online literature has become a trend, spreading positive messages about China and enhancing the reach and influence of Chinese culture. Carrying the genes of traditional Chinese culture, online literature utilizes digital technology and internet platforms to showcase the fusion of traditional Chinese culture and cutting-edge technology, demonstrating the vibrant charm of Chinese cultural innovation and playing an increasingly important role in disseminating Chinese culture and shaping China's image.

[0180] Since current online novels are highly popular among overseas readers, many of them are translated by human translators and published on Internet websites. As a result, there is a certain scale of analogous corpus on the Internet, but it is not a parallel corpus. Therefore, it cannot be directly used to train a machine translation model. It needs to be sentence-aligned between Chinese and English novels through an alignment tool before it can be directly used to train a machine translation model.

[0181] When performing the sentence automatic alignment algorithm, as shown in the bold part of Table 1, it can be found that due to algorithm errors, there will be cases where some sentences are misaligned, that is, the Chinese and English bilinguals are not fully translated into each other. For example, in Case 1, the translation of the Chinese sentence "I don't have the mind to care anymore" was omitted during alignment on the English side; in Case 2, the translation of the English sentence "However, he couldn't watch as she got married to someone else" was omitted during automatic alignment on the Chinese side.

[0182] Table 1 Omitted translation cases in automatic alignment

[0183]

[0184] After statistics, such error situations are shown in Table 2, accounting for about 32.5%:

[0185] Table 2 Statistical table of error ratios

[0186] 80 26 32.5％

[0187] When there are serious incomplete alignment situations in the training corpus, it will seriously affect the translation performance of the training model, resulting in missed translations in the translation model.

[0188] In the general domain, because there is high-quality aligned corpus, there is basically no problem of missed translation. Therefore, it is very necessary to improve the quality of aligned sentences, which can alleviate the problem of missed translation in the final translation model.

[0189] In related technologies, when sentence-aligning the original novel chapters, the following problems will occur:

[0190] 1) Sentence segmentation usually brings incomplete problems to sentences, and the sentence segmentation preprocessing causes some aligned corpus sentences to be incomplete;

[0191] 2) Because there are deviations in the translation field, the translation of the source sentence (the sentence in the source language) is inaccurate, and finally the alignment algorithm deviation causes misalignment and missed alignment problems in the corpus.

[0192] Therefore, the above problems will all lead to poor quality of aligned bilinguals, and the low-quality aligned bilingual corpus will directly affect the translation quality of the final translation model.

[0193] To address the aforementioned issues, this application provides a parallel corpus construction method and a text translation method. Based on paragraph information from the original sentences, this method uses recall to resolve the problem of incomplete sentences in partially aligned corpora caused by sentence segmentation preprocessing. It also alleviates misalignment and omissions in the corpus due to alignment algorithm biases, thereby improving the quality of the parallel corpus and ultimately enhancing translation quality. The automatic alignment is performed at the sentence level, incorporating paragraph boundary information specific to literary texts. In literary writing, authors write in paragraphs; using paragraphs as boundary information helps the algorithm recall some sentences, ultimately mitigating omissions in translation.

[0194] like Figure 8 As shown (taking Chinese and English novels as examples), firstly, it is assumed that after the original Chinese (source language) paragraph is punctuated, there are two Chinese clauses (Chinese clause 1 and Chinese clause 2). According to the alignment algorithm, the corresponding English (translation language) aligned sentence indexes are obtained (Chinese clause 1 corresponds to English clause 1 and English clause 2, and Chinese clause 2 corresponds to English clause 4 and English clause 5). Finally, using the original paragraph as boundary information, the clauses on the Chinese side are merged to obtain aligned Chinese paragraphs, and the aligned clauses on the English side are merged to obtain aligned English paragraphs (which can recall some previously misaligned English clauses, such as English clause 3). Finally, the aligned Chinese paragraphs and aligned English paragraphs form a paragraph-level bilingual corpus. The translation model is trained through the bilingual corpus, thereby improving the translation performance of the translation model.

[0195] Because some Chinese sentences form separate paragraphs, paragraph boundary information cannot recall missing clauses for misaligned sentences. When finding relevant Chinese and English paragraphs based on the original Chinese paragraph information, misalignment can also occur in the Chinese paragraphs, leading to misalignment of the first or last sentence of the final merged English paragraph. This is a legacy issue of misalignment. Based on this, this application's embodiments further refine the handling of misalignment. During automatic alignment, the presence of blank spaces at either the source (Chinese) or target (English) end indicates a high probability of misalignment in the sentences before and after that point. In other words, blank spaces are a key signal of misalignment. Therefore, the precise misalignment repair solution is to locate the blank sentence position and, based on this position, simultaneously merge the sentences at both the source and target ends to construct a pseudo-paragraph, thereby recalling the misaligned sentence to this pseudo-paragraph.

[0196] like Figure 9 As shown, since the source or target end finds the Chinese clause i in the empty space, the English first sentence alignment clause 1 corresponding to the Chinese clause i-1 is obtained according to the automatic alignment algorithm. i-1The Chinese clause i+1 corresponds to the English side clause n. i +1 Then, based on the information of the first and last sentences on both the Chinese and English sides, the sentences on both sides are merged, which can recall the missing Chinese clause i.

[0197] like Figure 10 As shown, the specific steps of the parallel corpus construction method provided in this application embodiment are as follows:

[0198] Step 11: Extract paragraph information from the original novel chapter data to obtain a chapter alignment list.

[0199] Step 12: Use the new chapter alignment list to align the sentences of the original chapter data to obtain bilingual Chinese and English novel paragraphs.

[0200] Step 13: Train the translation model using bilingual Chinese and English novel passages.

[0201] Step 14: Perform the translation task using the trained translation model.

[0202] like Figure 11 As shown, assume a Chinese paragraph contains n clauses. Using Python, we first locate the first Chinese clause (1) of the original Chinese paragraph, and then match its aligned English clause (1) based on the index. 1 The index matching is performed sequentially according to the sentence breaks in the Chinese paragraph, until the index of the last Chinese clause n of the original Chinese paragraph matches the aligned last English clause n. n , in English clause 1 1 English clause n n To establish beginning and end boundary information, the English aligned clauses in between are merged to form aligned English paragraphs. This results in the original Chinese paragraphs and the aligned English paragraphs forming a paragraph-level bilingual corpus. Based on the record list generated from sentence segmentation (Chinese clause 1 to Chinese clause n) and the record list generated from alignment (alignment relationship between Chinese and English clauses), the Chinese and English clauses belonging to the same paragraph are traced and merged into a paragraph-level bilingual corpus.

[0203] It should be noted that the parallel corpus construction method provided in this application includes two modules: a paragraph information extraction module and a sentence alignment module. The paragraph information extraction module and the sentence alignment module are described below with reference to the figures:

[0204] The following is an explanation of the paragraph information extraction module:

[0205] Since the sentence segmentation tool and alignment tool only output the processed text data, in order to recall the clauses of an original paragraph and its aligned sentences, it is necessary to track the alignment of the clauses after the original paragraph is segmented. In this application embodiment, the sentence segmentation tool and alignment tool are improved by saving and outputting the sentence segmentation operation records in the form of a list to obtain the chapter sentence segmentation list, and saving and outputting the alignment operation records in the form of a list to obtain the chapter alignment list.

[0206] like Figure 12 As shown, Chinese paragraphs undergo preprocessing (e.g., sentence segmentation, illegal character handling) to obtain Chinese clauses, and English paragraphs undergo preprocessing (e.g., sentence segmentation, illegal character handling) to obtain English clauses. An alignment tool is then used to align the Chinese and English clauses, and the alignment records are saved to obtain a chapter alignment list. Specifically, the sentence segmentation tool is used to segment the Chinese paragraphs, and the segmentation records are saved to obtain a chapter segmentation list.

[0207] After obtaining the chapter segmentation list and the chapter alignment list, the chapter segmentation list and the chapter alignment list are reorganized to obtain an alignment mapping list containing paragraph information (including Chinese clause identifiers containing paragraph information and English clause identifiers containing paragraph information), which provides guidance information for the next step of alignment.

[0208] The following is an explanation of the sentence alignment module:

[0209] like Figure 13 As shown, after obtaining the alignment mapping list based on the paragraph information, sentences are extracted from the preprocessed Chinese and English chapters based on the Chinese and English clause identifiers in the alignment mapping list, and the extracted clauses are merged to finally obtain bilingual Chinese and English novel paragraphs.

[0210] The following examples illustrate the beneficial effects of the parallel corpus construction method provided in this application:

[0211] 1) Regarding the issue of correcting sentence breaks

[0212] Before automatic alignment, data preprocessing is required. To improve the quality of automatic alignment, sentences are generally segmented as short as possible. Therefore, when segmenting sentences with quotation marks (dialogue), sentences with ending characters within the quotation marks are also segmented. However, this can lead to incomplete bilingual sentences after alignment. Novels contain a large number of sentences with quotation marks (dialogue). The method provided in this application can solve this problem, as shown in Table 3:

[0213] Table 3 Case Studies of Sentence Segmentation Problem Correction

[0214]

[0215]

[0216] After random inspection, the proportion of sentence segmentation problems in the original aligned bilingual corpus and the repair rate of this problem in the embodiments of this application are shown in Table 4:

[0217] Table 4 Repair situation table of sentence segmentation problems

[0218] 16.25％(13 / 80) 100％(13 / 13)

[0219] As can be seen from Table 4, after repair, the sentences in the paragraph version retain more of the original information of the dialogue than the sentences in the aligned version.

[0220] 2) Repair alignment problems according to real paragraphs

[0221] Due to the error problem of the alignment algorithm, there are missing alignments in the bilinguals directly aligned automatically. By effectively using the paragraph boundary information in the embodiments of this application, some missing-aligned sentences are recalled. Specific cases are shown in Table 5, where the bold font is the part missing-aligned by another source end:

[0222] Table 5 Cases of repairing alignment problems

[0223]

[0224]

[0225] Among them, Case 1 and Case 2 respectively represent two common types of missing-alignment situations during sentence alignment: missing-alignment on the Chinese side (for example, the translation of "I have no intention to care anymore" is missing on the English side), and missing-alignment on the English side (for example, the translation of "However, he couldn’t watch as she got married to someone else" is missing on the Chinese side).)

[0226] After random inspection, the proportion of missing-alignment problems in the original aligned bilingual corpus and the repair rate of this work for this problem are shown in Table 6:

[0227] Table 6 Repair situation table of missing-alignment

[0228] 32.5％(26 / 80) 46.1％(12 / 26)

[0229] 3) Repair alignment problems according to pseudo-paragraphs

[0230] Because there are some Chinese sentences that form paragraphs alone, or there are missing alignments in the first or last sentences of Chinese paragraphs, such missing-alignment problems cannot be repaired based on the real paragraph boundaries.

[0231] By further locating the misaligned positions and merging the upper and lower parts to construct pseudo-paragraphs, the boundary information can be effectively utilized to recall the misaligned sentences. A specific case is shown in Table 7, where the bold font indicates the part misaligned by another source:

[0232] Table 7 Cases of Repairing Alignment Issues

[0233]

[0234] As can be seen from Table 8, after the merger of pseudo-paragraphs, the corresponding translation of the original Chinese sentence "She climbed up his body and grasped him like he was a life-saving straw." was recalled.

[0235] After random inspection, the further repair situation of the pseudo-paragraph alignment repair based on the real paragraph repair is shown in Table 8:

[0236] Table 8 Table of Missing Alignment Repair Situations

[0237] 17.5％(14 / 80) 64.3％(9 / 14) <s

[0238] 4) Improve the performance of the translation model

[0239] The embodiments of the present application can greatly improve the quality of automatically aligned corpora, and increase the paragraph information, ultimately improving the data quality of the bilingual corpus, thereby improving the performance of the translation model in the field of online novels and significantly enhancing the user experience. Among them, the embodiments of the present application improve the data alignment quality and expand the bilingual quantity by embedding the automatic alignment work of online novels based on self-training, and further optimize the model training performance.

[0240] In summary, the parallel corpus construction method provided by the embodiments of the present application realigns and corrects the preprocessed novel chapters according to the original chapters containing paragraph information, and has the following beneficial effects:

[0241] 1) Solve the problem that some aligned corpus sentences are incomplete due to sentence segmentation preprocessing, and effectively recall some missing-aligned sentences caused by alignment algorithm errors, so as to obtain more high-quality aligned data, alleviate the problem of missing translations of the model, and then improve the performance of the translation model through accurate bilingual corpora;

[0242] 2) Introduce data with boundary information by merging to increase the number of training corpora, help the model better translate sentences with paragraph information, and then improve the performance of the translation model.

[0243] The parallel corpus construction method and text translation method provided in this application have been described above in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application. This application also provides a parallel corpus construction apparatus and a text translation apparatus. In practical applications, the functional modules in the parallel corpus construction apparatus and the text translation apparatus can be collaboratively implemented by the hardware resources of the electronic device (such as a terminal device, server, or server cluster), such as computing resources like processors, communication resources (such as those used to support various communication methods like optical fiber and cellular), and memory. Figure 4 The parallel corpus construction device 555 and the text translation device stored in the memory 550 are shown. They can be software in the form of programs and plug-ins, such as software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules, application programming interfaces, plug-ins, cloud services, etc. in large software systems. Examples of different implementation methods are given below.

[0244] The parallel corpus construction device 555 includes a series of modules, including a first acquisition module 5551, an alignment module 5552, a merging module 5553, a construction module 5554, a training module 5555, and a processing module 5556. The following description further illustrates how the various modules in the parallel corpus construction device 555 provided in this embodiment cooperate to implement the parallel corpus construction scheme.

[0245] The first acquisition module 5551 is used to acquire multiple first sentences of a first text in a first language and multiple second sentences of a second text in a second language; the alignment module 5552 is used to perform alignment query processing on multiple second sentences of the second text based on each first sentence, and to take the second sentence that is successfully aligned with any first sentence as the aligned sentence; and to take the second sentence that matches the boundary information of the first text and fails to align with all of the multiple first sentences as the missing aligned sentence; the merging module 5553 is used to merge the aligned sentence and the missing aligned sentence based on the boundary information of the first text to obtain a third text aligned with the first text; and the construction module 5554 is used to construct a bilingual parallel corpus based on the first text and the third text.

[0246] In some embodiments, the alignment module 5552 is further configured to obtain an alignment list between the first text and the second text; wherein the alignment list includes alignment mapping relationships between sentences in the first text and sentences in the second text; and to query the alignment list based on any first sentence in the first text, and take the queried sentence in the second text as the aligned sentence.

[0247] In some embodiments, the alignment module 5552 is further configured to perform text translation processing on any first sentence of the first text using the text translation model to obtain a translated sentence of any first sentence; match the translated sentence with the plurality of second sentences respectively, and take the successfully matched second sentence as the second sentence aligned with any first sentence; and construct an alignment list between the first text and the second text based on the second sentence aligned with any first sentence.

[0248] In some embodiments, the alignment module 5552 is further configured to perform text encoding processing on the translated sentence to obtain the encoding vector of the translated sentence; perform text encoding processing on the plurality of second sentences respectively to obtain the encoding vector of each second sentence; perform similarity processing on the encoding vector of the translated sentence and the encoding vector of each second sentence to obtain the similarity between the translated sentence and each second sentence respectively; when any of the similarities is greater than the similarity threshold, the second sentence corresponding to any of the similarities is taken as the second sentence aligned with any of the first sentences.

[0249] In some embodiments, the alignment module 5552 is further configured to perform heuristic alignment processing on the translated sentence and any of the second sentences to obtain a first alignment result of any of the second sentences; perform length alignment processing on the translated sentence and any of the second sentences to obtain a second alignment result of any of the second sentences; and when both the first alignment result and the second alignment result indicate successful alignment, the second sentence is regarded as the second sentence aligned with any of the first sentences.

[0250] In some embodiments, the alignment module 5552 is further configured to obtain punctuation marks in the translated sentence and obtain punctuation marks in any of the second sentences; compare the punctuation marks in the translated sentence with the punctuation marks in any of the second sentences, and use the comparison result as the first alignment result of any of the second sentences.

[0251] In some embodiments, the alignment module 5552 is further configured to obtain the sentence length of the translated sentence and the sentence length of any second sentence; determine a first ratio between the sentence length of the translated sentence and the sentence length of any second sentence; perform matching processing on the first ratio and a set ratio range, and use the matching result as the second alignment result of any second sentence.

[0252] In some embodiments, the apparatus further includes: a processing module 5556, configured to obtain a sentence segmentation list of the first text; wherein the sentence segmentation list includes the sentence segmentation order of each of the first sentences; based on the sentence segmentation list, determine the first first sentence and the first last sentence in the first text, and use the first first sentence and the first last sentence as boundary information of the first text; use the second sentence in the second text corresponding to the first first sentence as the second first sentence in the second text, and the second sentence corresponding to the first last sentence as the second last sentence in the second text; use the second sentence in the second text located between the second first sentence and the second last sentence as the second sentence that matches the boundary information of the first text.

[0253] In some embodiments, the processing module 5556 is further configured to perform sentence segmentation processing on the first text based on set punctuation marks to obtain multiple first sentences of the first text, and record the sentence segmentation order of each first sentence based on the sentence segmentation processing; and construct a sentence segmentation list of the first text based on the sentence segmentation order of each first sentence.

[0254] In some embodiments, the merging module 5553 is further configured to take the aligned sentences that match the boundary information of the first text as sentences to be merged; and to merge the sentences to be merged and the missing aligned sentences in the order of the sentences to be merged and the missing aligned sentences to obtain a third text aligned with the first text.

[0255] In the above technical solution, the first text includes a first paragraph and a second paragraph adjacent to the first paragraph. The first paragraph is a first sentence, and the second paragraph includes at least one first sentence. The alignment module 5552 is further configured to merge the first paragraph and the second paragraph to obtain a pseudo paragraph of the first text when the first paragraph fails to align with the plurality of second sentences. The boundary information of the pseudo paragraph is the boundary information of the first text. The construction module 5554 is further configured to construct a bilingual parallel corpus based on the pseudo paragraph and the third text.

[0256] In some embodiments, the apparatus further includes: a training module 5555, configured to perform text translation processing on the first text using a text translation model to obtain translated text of the first text; construct a loss function of the text translation model based on the translated text of the first text and the third text; update the parameters of the text translation model until the loss function converges; and use the updated parameters of the text translation model when the loss function converges as the parameters of the trained text translation model.

[0257] In some embodiments, the training module 5555 is further configured to perform the following processing through the text translation model: encode the first text to obtain an encoding vector of the first text; decode the encoding vector of the first text to generate a translation vector corresponding to the first text; and combine the translation words corresponding to the translation vectors based on the order of the generated translation vectors to obtain the translated text of the first text.

[0258] The text translation device includes a series of modules, including a second acquisition module and a translation module. The following description further explains the scheme by which the various modules in the text translation device provided in this application cooperate to achieve text translation.

[0259] The second acquisition module is used to acquire the text to be translated in the first language; the translation module is used to perform text translation processing on the text to be translated using the text translation model to obtain the translated text of the text to be translated.

[0260] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the parallel corpus construction method or text translation method described in this application.

[0261] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the parallel corpus construction method or text translation method provided in this application. For example, ... Figures 5A-5B The method for constructing parallel corpora is shown.

[0262] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0263] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0264] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0265] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0266] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for constructing parallel corpora, characterized in that, The method includes: Get multiple first sentences from the first text in the first language, and get multiple second sentences from the second text in the second language; Alignment query processing is performed on multiple second sentences of the second text based on each first sentence, and the second sentence that is successfully aligned with any first sentence is taken as the aligned sentence; Obtain a list of sentence segments for the first text; based on the list of sentence segments, determine the first first sentence and the first last sentence in the first text; and use the first first sentence and the first last sentence as boundary information of the first text. The list of sentence segments includes the sentence segmentation order of each first sentence. The second sentence in the second text that corresponds to the first sentence is taken as the second first sentence in the second text, and the second sentence that corresponds to the first last sentence is taken as the second last sentence in the second text; The second sentence in the second text, located between the second first sentence and the second last sentence, is taken as the second sentence that matches the boundary information of the first text; When the first text includes a first paragraph and a second paragraph adjacent to the first paragraph, the first paragraph is a first sentence, the second paragraph includes at least one first sentence, and the first paragraph fails to align with the plurality of second sentences, the first paragraph and the second paragraph are merged to obtain a pseudo paragraph of the first text, wherein the boundary information of the pseudo paragraph is the boundary information of the first text; When the first text includes a third paragraph and a fourth paragraph adjacent to the third paragraph, and the first or last sentence of the third paragraph fails to align with the plurality of second sentences, the third paragraph and the fourth paragraph are merged to construct a pseudo paragraph of the first text, and the first and last sentences of the pseudo paragraph are used as the boundary information of the first text. The second sentence that matches the boundary information of the first text and fails to align with all of the first sentences is considered a missing alignment sentence. Based on the boundary information of the first text, the aligned sentence and the missing aligned sentence are merged to obtain a third text aligned with the first text; A bilingual parallel corpus is constructed based on the first text and the third text.

2. The method according to claim 1, characterized in that, The step of performing alignment query processing on multiple second sentences of the second text based on each first sentence, and taking the second sentence that is successfully aligned with any first sentence as the aligned sentence, includes: Obtain the alignment list between the first text and the second text; The alignment list includes the alignment mapping relationship between sentences in the first text and sentences in the second text; Based on any of the first sentences in the first text, query the alignment list and use the sentences in the second text that are found in the query as the alignment sentences.

3. The method according to claim 2, characterized in that, The step of obtaining the alignment list between the first text and the second text includes: The first sentence of the first text is translated using a text translation model to obtain the translated sentence of any first sentence. The translated sentence is matched with each of the plurality of second sentences, and the second sentence that is successfully matched is taken as the second sentence that is aligned with any of the first sentences; Based on a second sentence that aligns with any of the first sentences, construct an alignment list between the first text and the second text.

4. The method according to claim 3, characterized in that, The step of matching the translated sentence with each of the plurality of second sentences, and taking the successfully matched second sentence as the second sentence aligned with any of the first sentences, includes: The translated sentence is processed by text encoding to obtain the encoded vector of the translated sentence; The multiple second sentences are processed by text encoding to obtain the encoding vector of each second sentence; The similarity between the encoding vector of the translated sentence and the encoding vector of each second sentence is calculated to obtain the similarity between the translated sentence and each second sentence. When any of the aforementioned similarities is greater than the similarity threshold, the second sentence corresponding to any of the aforementioned similarities is taken as the second sentence aligned with any of the first sentences.

5. The method according to claim 3, characterized in that, The step of matching the translated sentence with each of the plurality of second sentences, and taking the successfully matched second sentence as the second sentence aligned with any of the first sentences, includes: The translated sentence is heuristically aligned with any of the second sentences to obtain a first alignment result for any of the second sentences; The translated sentence is length-aligned with any of the second sentences to obtain a second alignment result for any of the second sentences. When both the first alignment result and the second alignment result indicate successful alignment, any second sentence is taken as the second sentence aligned with any first sentence.

6. The method according to claim 5, characterized in that, The step of performing heuristic alignment processing on the translated sentence and any of the second sentences to obtain a first alignment result for any of the second sentences includes: Obtain the punctuation marks in the translated sentence, and obtain the punctuation marks in any of the second sentences; The punctuation marks in the translated sentence are compared with the punctuation marks in any of the second sentences, and the comparison result is used as the first alignment result of any of the second sentences.

7. The method according to claim 5, characterized in that, The length alignment process between the translated sentence and any of the second sentences to obtain a second alignment result for any of the second sentences includes: Obtain the sentence length of the translated sentence, and obtain the sentence length of any of the second sentences; Determine a first ratio between the sentence length of the translated sentence and the sentence length of any of the second sentences; The first ratio is matched with a set ratio range, and the matching result is used as the second alignment result for any second sentence.

8. The method according to claim 1, characterized in that, The step of obtaining the sentence segmentation list of the first text includes: The first text is segmented based on the set punctuation marks to obtain multiple first sentences of the first text, and the segmentation order of each first sentence is recorded based on the segmentation process. Based on the sentence segmentation order of each of the first sentences, construct a sentence segmentation list for the first text.

9. The method according to claim 1, characterized in that, The step of merging the aligned sentence and the missing aligned sentence based on the boundary information of the first text to obtain a third text aligned with the first text includes: The aligned sentences that match the boundary information of the first text are taken as the sentences to be merged; According to the order of the sentences to be merged and the missing alignment sentences, the sentences to be merged and the missing alignment sentences are merged to obtain a third text that is aligned with the first text.

10. The method according to claim 1, characterized in that, The method further includes: The first text is translated using a text translation model to obtain the translated text of the first text; Based on the translated text of the first text and the third text, construct the loss function of the text translation model; Update the parameters of the text translation model until the loss function converges, and use the updated parameters of the text translation model when the loss function converges as the parameters of the trained text translation model.

11. A parallel corpus construction device, characterized in that, The device includes: The first acquisition module is used to acquire multiple first sentences of the first text in the first language and multiple second sentences of the second text in the second language; The alignment module is used to perform alignment query processing on multiple second sentences of the second text based on each first sentence, and to take the second sentence that is successfully aligned with any of the first sentences as the aligned sentence; The processing module is configured to obtain a list of sentence segments in the first text, determine the first first sentence and the first last sentence in the first text based on the sentence segment list, and use the first first sentence and the first last sentence as boundary information of the first text, wherein the sentence segment list includes the sentence segmentation order of each first sentence; the second sentence in the second text corresponding to the first first sentence is used as the second first sentence in the second text, and the second sentence corresponding to the first last sentence is used as the second last sentence in the second text; the second sentence in the second text located between the second first sentence and the second last sentence is used as the second sentence that matches the boundary information of the first text. The alignment module is configured to: when the first text includes a first paragraph and a second paragraph adjacent to the first paragraph, where the first paragraph is a first sentence and the second paragraph includes at least one first sentence, and the first paragraph fails to align with the plurality of second sentences, merge the first paragraph and the second paragraph to obtain a pseudo-paragraph of the first text, wherein the boundary information of the pseudo-paragraph is the boundary information of the first text; when the first text includes a third paragraph and a fourth paragraph adjacent to the third paragraph, and the first or last sentence of the third paragraph fails to align with the plurality of second sentences, merge the third paragraph and the fourth paragraph to construct a pseudo-paragraph of the first text, using the first and last sentences of the pseudo-paragraph as the boundary information of the first text; and designate second sentences that match the boundary information of the first text and fail to align with the plurality of first sentences as missing alignment sentences; The merging module is used to merge the aligned sentence and the missing aligned sentence based on the boundary information of the first text to obtain a third text aligned with the first text. A construction module is used to construct a bilingual parallel corpus based on the first text and the third text.

12. The apparatus according to claim 11, characterized in that, The alignment module is also used for: Obtain the alignment list between the first text and the second text; The alignment list includes the alignment mapping relationship between sentences in the first text and sentences in the second text; Based on any of the first sentences in the first text, query the alignment list and use the sentences in the second text that are found in the query as the alignment sentences.

13. The apparatus according to claim 12, characterized in that, The alignment module is also used for: The first sentence of the first text is translated using a text translation model to obtain the translated sentence of any first sentence. The translated sentence is matched with each of the plurality of second sentences, and the second sentence that is successfully matched is taken as the second sentence that is aligned with any of the first sentences; Based on a second sentence that aligns with any of the first sentences, construct an alignment list between the first text and the second text.

14. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the parallel corpus construction method according to any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the parallel corpus construction method according to any one of claims 1 to 10 when executed by a processor.

16. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the parallel corpus construction method according to any one of claims 1 to 10.

Images