Parallel corpus processing method, device, equipment and computer readable storage medium

Abstract

The present application provides a parallel corpus processing method, apparatus, device and computer-readable storage medium. The method performs sentence segmentation operation on a target parallel corpus to obtain M original texts of an original document and N translated texts of a translated document in the target parallel corpus; encodes the M original texts and the N translated texts to obtain a vector corresponding to each original text and a vector corresponding to each translated text; performs segmentation operation on the target parallel corpus according to the obtained vectors to obtain multiple bilingual translation segments; performs alignment operation on each bilingual translation segment to obtain the alignment result of the target parallel corpus, thereby reducing the workload and improving the accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of translation technology, and in particular to a parallel corpus processing method, device, equipment and computer-readable storage medium. Background Art

[0002] Parallel corpus is a bilingual corpus consisting of the original text and its parallel target text. Parallel corpus is an important resource for training machine translation models. It plays an indispensable role in translation conversion research, translation style research and filling the gaps in bilingual dictionaries.

[0003] In the related art, a common method for processing parallel corpora of long documents is to use manually formulated rules to screen and construct feature words and phrases that can represent the similarity between two sentences, use these word features to calculate the similarity between the two sentences, and determine the alignment result of the parallel corpus based on the similarity.

[0004] However, the above method can only align single sentences, and cannot accurately handle the situation of multiple sentences. In addition, the accuracy is low, the workload is large, and the efficiency is low when processing long documents. Summary of the invention

[0005] The present application provides a parallel corpus processing method, device, equipment and computer-readable storage medium, thereby solving the technical problems that the prior art can only align single sentences, cannot accurately process the situation of multiple sentences, and has low accuracy, large workload and low efficiency when processing long documents.

[0006] In a first aspect, the present application provides a parallel corpus processing method, comprising:

[0007] Performing sentence segmentation operation on the target parallel corpus to obtain M original sentences of the original document and N translated sentences of the translated document in the target parallel corpus;

[0008] Encoding the M original texts and the N translated texts to obtain a vector corresponding to each original text and a vector corresponding to each translated text;

[0009] According to the obtained vectors, the target parallel corpus is segmented to obtain a plurality of bilingual translation segments;

[0010] An alignment operation is performed on each bilingual translation segment to obtain an alignment result of the target parallel corpus.

[0011] Here, the embodiment of the present application first performs sentence segmentation and encoding operations on the target parallel corpus, so that accurate similarity calculation can be performed for each short sentence, sentence pairs that can be used to segment the document are found, the entire document is segmented to obtain multiple short documents, and the alignment results of the short documents are obtained by splicing the sentences of the short documents and calculating the similarity. Since the embodiment of the present application can segment a long document into short documents, the alignment of complex long documents can be converted into that of short documents and the sentences can be spliced ​​and then aligned, the problems of poor accuracy of aligning long documents in the prior art and the inability to accurately handle multi-sentence pairs and multi-sentences are solved, and the accuracy and efficiency of the alignment of the target parallel corpus are improved.

[0012] Optionally, segmenting the target parallel corpus according to the obtained vector to obtain a plurality of bilingual translation segments includes:

[0013] According to the obtained vectors, the first similarity of M*N sentence pairs consisting of any original sentence in the M original sentences and any translated sentence in the N translated sentences is calculated;

[0014] Determining sentence pairs for segmenting the document according to a first preset rule and the first similarity;

[0015] The target parallel corpus is segmented according to the sentence pairs of the segmented documents to obtain the multiple bilingual translation segments.

[0016] The embodiment of the present application calculates the similarity between all original texts and all translations to determine the sentence pairs, i.e., segmentation points, for segmenting the document, and segments the target mutual translation document according to the sentence pairs of the segmented document. Here, by calculating the similarity between all original texts and all translations, the original texts and translations with the highest similarity can be accurately determined, thereby accurately dividing the target mutual translation segments, further improving the accuracy of the target parallel corpus alignment.

[0017] Optionally, performing an alignment operation on each bilingual translation segment to obtain an alignment result of the target parallel corpus includes:

[0018] splicing the plurality of bilingual translation segments into a plurality of sentence combinations according to the order of the target parallel corpus;

[0019] Encoding the multiple sentence combinations to obtain a vector corresponding to each combination;

[0020] Calculating the second similarity between the original text and the translation in the plurality of sentence combinations according to the obtained vectors;

[0021] According to the second preset rule and the second similarity, an alignment result of the target parallel corpus is obtained.

[0022] Here, the embodiment of the present application splices the split short documents according to the original order of the target parallel corpus to obtain multiple sentence combinations, and then calculates the similarity of all the spliced ​​sentence combinations to obtain the sentence combination with the highest similarity between the original text and the translation, thereby achieving accurate alignment and further improving the accuracy of the alignment of the target parallel corpus.

[0023] Optionally, determining sentence pairs for segmenting the document according to a first preset rule and the first similarity includes:

[0024] Deleting sentence pairs whose first similarity is lower than a first preset similarity threshold from the M*N sentence pairs, and screening the remaining sentence pairs in the M*N sentence pairs according to the first preset rule;

[0025] According to the screening result, sentence pairs of the segmented document are determined.

[0026] Here, the embodiment of the present application deletes sentence pairs whose first similarity is lower than the first preset similarity threshold. At the same time, according to the first preset rule, sentence pairs suitable as positioning points can be screened out, the segmentation points for segmenting the documents can be accurately determined, and the target parallel corpus can be accurately and finely divided, thereby further improving the accuracy of the alignment of the target parallel corpus.

[0027] Optionally, before screening the remaining sentence pairs in the M*N sentence pairs, the method further includes:

[0028] determining relations between sentences in the remaining sentence pairs;

[0029] If it is determined based on the relationship that the similarity of any sentence in the to-be-processed sentence pair in the remaining sentence pairs is lower than the similarity of the same sentence in other sentence pairs, the to-be-processed sentence pair is deleted.

[0030] Here, before screening the remaining sentence pairs among the M*N sentence pairs, the embodiment of the present application determines the relationship between the sentence pairs and screens the sentence pairs to facilitate determining the sentence pairs used to segment the document among the remaining sentence pairs. In addition, if any sentence in a sentence pair appears in a sentence pair with a higher score than it, then it can be determined that the other sentence has a higher match degree than the current sentence pair, and then the current sentence pair is deleted.

[0031] Optionally, before screening the remaining sentence pairs in the M*N sentence pairs, the method further includes:

[0032] If the ratio of the number of words to the number of characters in a sentence pair exceeds a first preset ratio threshold, the sentence pair is deleted;

[0033] or,

[0034] If the ratio of the lengths of the segments of a sentence pair after being segmented according to the labels exceeds a second preset ratio threshold, the sentence pair is deleted;

[0035] or,

[0036] Concatenate any sentence in a sentence pair with its previous sentence or next sentence and recalculate the similarity score of the modified sentence pair. If the similarity score of the new sentence pair becomes higher, delete the sentence pair.

[0037] or,

[0038] If the General Language Understanding Evaluation (GLEU) score of a sentence pair after translation into the same language is lower than the preset score threshold, the sentence pair is deleted;

[0039] or,

[0040] If the number of intersections between a sentence pair and other sentence pairs is greater than the preset number of intersections, the sentence pair is deleted.

[0041] Here, before screening the remaining sentence pairs in the M*N sentence pairs, the embodiment of the present application provides several methods for screening sentence pairs. The sentence pairs are selected based on the ratio of the number of words or characters in the sentence pairs, the ratio of the length of the segments after segmentation by labels, the similarity score, the GLEU score and the number of intersections between the original text and the translation. Sentence pairs that are not suitable as the target parallel corpus for separation are pre-screened and deleted, thereby reducing the workload of subsequently determining the sentence pairs for segmenting the document and further improving the efficiency and accuracy of segmentation and alignment.

[0042] In a second aspect, an embodiment of the present application provides a parallel corpus processing device, comprising:

[0043] A sentence segmentation module, used to perform a sentence segmentation operation on the target parallel corpus to obtain M original sentences of the original document and N translated sentences of the translated document in the target parallel corpus;

[0044] An encoding module, used for encoding the M original sentences and the N translated sentences to obtain a vector corresponding to each original sentence and a vector corresponding to each translated sentence;

[0045] A segmentation module, used for segmenting the target parallel corpus according to the obtained vectors to obtain multiple bilingual translation segments;

[0046] The alignment module is used to perform an alignment operation on each bilingual translation segment to obtain an alignment result of the target parallel corpus.

[0047] Optionally, the segmentation module is specifically used for:

[0048] According to the obtained vectors, the first similarity of M*N sentence pairs consisting of any original sentence in the M original sentences and any translated sentence in the N translated sentences is calculated;

[0049] Determining sentence pairs for segmenting the document according to a first preset rule and the first similarity;

[0050] The target parallel corpus is segmented according to the sentence pairs of the segmented documents to obtain the multiple bilingual translation segments.

[0051] Optionally, the alignment module is specifically used for:

[0052] splicing the plurality of bilingual translation segments into a plurality of sentence combinations according to the order of the target parallel corpus;

[0053] Encoding the multiple sentence combinations to obtain a vector corresponding to each combination;

[0054] Calculating the second similarity between the original text and the translation in the plurality of sentence combinations according to the obtained vectors;

[0055] According to the second preset rule and the second similarity, an alignment result of the target parallel corpus is obtained.

[0056] Optionally, the segmentation module is specifically used for:

[0057] Deleting sentence pairs whose first similarity is lower than a first preset similarity threshold from the M*N sentence pairs, and screening the remaining sentence pairs in the M*N sentence pairs according to the first preset rule;

[0058] According to the screening result, sentence pairs of the segmented document are determined.

[0059] Optionally, before screening the remaining sentence pairs in the M*N sentence pairs, the segmentation module is further used to:

[0060] determining relations between sentences in the remaining sentence pairs;

[0061] If it is determined based on the relationship that the similarity of any sentence in the to-be-processed sentence pair in the remaining sentence pairs is lower than the similarity of the same sentence in other sentence pairs, the to-be-processed sentence pair is deleted.

[0062] Optionally, before screening the remaining sentence pairs in the M*N sentence pairs, the segmentation module is further used to:

[0063] If the ratio of the number of words to the number of characters in a sentence pair exceeds a first preset ratio threshold, the sentence pair is deleted;

[0064] or,

[0065] If the ratio of the lengths of the segments of a sentence pair after being segmented according to the labels exceeds a second preset ratio threshold, the sentence pair is deleted;

[0066] or,

[0067] If any sentence in a sentence pair is concatenated with the previous or next sentence in the original text, and the similarity score of the new sentence pair becomes higher (the similarity score here can be the GLEU score after translation into the same language or the cosine similarity score after encoding by the encoder), then delete the sentence pair;

[0068] or,

[0069] If the natural language understanding benchmark GLEU score of the sentence pair after translation into the same language is lower than the first preset score, the sentence pair is deleted;

[0070] or,

[0071] If the number of intersections between a sentence pair and other sentence pairs is greater than the preset number of intersections, the sentence pair is deleted.

[0072] In a third aspect, an embodiment of the present application provides a parallel corpus processing device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the parallel corpus processing method as described in the first aspect or the optional manner of the first aspect.

[0073] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the parallel corpus processing method as described in the first aspect or the optional method of the first aspect.

[0074] The parallel corpus processing method, apparatus, device and computer-readable storage medium provided in the embodiments of the present application first perform sentence segmentation and encoding operations on the target parallel corpus, so that accurate similarity calculation can be performed for each short sentence, sentence pairs that can be used to segment the document are found, the entire document is segmented to obtain multiple short documents, and the alignment result of the long document is obtained by splicing and calculating the similarity of the short documents. Since the embodiments of the present application can segment long documents into short documents and divide the alignment of complex long documents into the splicing and alignment of short documents, the problems of slow speed and low accuracy in the prior art when aligning long documents for multi-sentence to multi-sentence are solved, and the accuracy and efficiency of the alignment of the target parallel corpus are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0076] Figure 1 A schematic diagram of a parallel corpus processing system architecture provided in an embodiment of the present application;

[0077] Figure 2 A flowchart of a parallel corpus processing method provided in an embodiment of the present application;

[0078] Figure 3 A flowchart of another parallel corpus processing method provided in an embodiment of the present application;

[0079] Figure 4 A schematic diagram of a segmentation process provided in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of an alignment process provided in an embodiment of the present application;

[0081] Figure 6 A schematic diagram of the structure of a parallel corpus processing device provided in an embodiment of the present application;

[0082] Figure 7 A schematic diagram of the structure of a parallel corpus processing device provided in an embodiment of the present application;

[0083] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0085] The terms "first", "second", "third", and "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0086] Parallel corpus is a bilingual or multilingual corpus consisting of the original text and its parallel corresponding target text. Parallel corpus is an important resource for training machine translation models. It plays an indispensable role in improving the quality of machine translation in the translation field, studying translation style, and filling the gaps in the meaning of bilingual dictionaries.

[0087] In the related art, the alignment of parallel corpora for long documents is usually divided into two steps. The first step is to design a method to calculate the similarity between two sentences, and the second step is to align them according to a specific algorithm based on the calculated similarity. Common means of calculating similarity include using artificially formulated rules to screen and construct characteristic words and phrases that can represent similarity in two sentences, and using these word features and some rules to calculate the similarity between the two sentences. However, the prior art requires manual construction of a one-to-one vocabulary, and there is currently no perfect rule for calculating similarity. It is often necessary to add "patches" to the rules for calculating similarity for different situations, which consumes work. In the alignment algorithm of the second step, the alignment algorithm based on dynamic programming, which is currently widely used, is slow when processing long documents, and because it requires in principle that the similarity is calculated for each sentence combination (including errors) and it will be taken into consideration when calculating the alignment result, it is difficult to ignore the poor quality and easy alignment error sentence pairs, resulting in the misaligned sentences caused by the poor quality of the sentence itself or the imperfect rule formulation and other error factors will also reduce the alignment quality of its neighboring sentences. The improved alignment scheme that uses neural networks to calculate similarity uses an encoder trained on multilingual data pairs based on tasks such as translation or classification to encode a sentence into a vector, and then calculates the similarity of the vectors of the two sentences through a certain algorithm. This avoids the incorrect scores caused by imperfect manual rules in the first step to a certain extent, and performs the final alignment according to some algorithms. The defect of this method is that if the dynamic programming algorithm is used in the second-step alignment algorithm, the quality problem mentioned above cannot be avoided. Some other alignment schemes that do not use dynamic programming cannot currently handle multi-sentence to multi-sentence situations conveniently. They can only align single sentences to single sentences, and then expand the single sentence alignment to multi-sentence alignment. The accuracy is not high enough. In addition, it does not consider the document order problem when aligning. As the document becomes longer and longer, the corresponding misalignment operations will become more frequent, which will also lead to low alignment accuracy.

[0088] In order to solve the above problems, the embodiments of the present application provide a parallel corpus processing method, device, equipment and computer-readable storage medium. When processing bilingual translation documents, the target parallel corpus is firstly segmented and encoded, so that accurate similarity calculation can be performed for each short sentence, sentence pairs that can be used to segment the document are found, and the entire document is segmented to obtain multiple short documents. The alignment result of the long document is obtained by splicing and calculating the similarity of the short documents. Since the embodiments of the present application can segment long documents into short documents and divide the alignment of complex long documents into the splicing and alignment of short documents, the problem that only single-sentence alignment can be performed on single-sentence in the existing non-dynamic programming algorithm scheme and multiple-sentence alignment cannot be accurately processed is solved. At the same time, because the similarity scores are sorted and screened, local errors caused by the dynamic programming algorithm when aligning sentences prone to errors can be avoided, thereby improving the accuracy and efficiency of the target parallel corpus alignment.

[0089] Optional, Figure 1 A schematic diagram of a parallel corpus processing system architecture provided in an embodiment of the present application. Figure 1 In the above architecture, the above architecture includes at least one of a receiving device 101, a processor 102 and a display device 103.

[0090] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the architecture of the parallel corpus processing system. In other feasible implementations of the present application, the above architecture may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0091] In a specific implementation process, the receiving device 101 may be an input / output interface or a communication interface.

[0092] When processing bilingual translation documents, the processor 102 can segment long documents into short documents, and then segment the alignment of complex long documents into the splicing alignment of short documents, accurately handle the problem of multi-sentence to multi-sentence, and do not need single-sentence alignment, thereby reducing the workload of alignment and improving the accuracy and efficiency of target parallel corpus alignment. Therefore, based on the aligned bilingual translation documents, translation conversion research, translation style research and filling in the missing meanings of bilingual dictionaries can be carried out in the translation field.

[0093] The display device 103 can be used to display the above results and the like.

[0094] The display device may also be a touch display screen, which is used to receive user instructions while displaying the above content to achieve interaction with the user.

[0095] It should be understood that the above-mentioned processor can be implemented by the processor reading instructions in the memory and executing the instructions, or it can be implemented by a chip circuit.

[0096] In addition, the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0097] The technical solution of the present application is described in detail below in conjunction with specific embodiments:

[0098] Figure 2 A flowchart of a parallel corpus processing method provided in an embodiment of the present application. The execution subject of the embodiment of the present application may be Figure 1 The specific execution subject of the processor 102 can be determined according to the actual application scenario. Figure 2 As shown, the method comprises the following steps:

[0099] S201: Perform sentence segmentation operation on the target parallel corpus to obtain M sentences of original text of the original document and N sentences of translated text of the translated document in the target parallel corpus.

[0100] Here, M and N may be the same or different.

[0101] Optionally, the sentence segmentation operation on the target parallel corpus can be based on punctuation marks in the target parallel corpus, or based on a preset number of words in each sentence, or based on certain specific words or phrases, and this application does not impose any specific restrictions on this.

[0102] S202: Encode the M original sentences and the N translated sentences to obtain a vector corresponding to each original sentence and a vector corresponding to each translated sentence.

[0103] The encoding here refers to obtaining a sentence vector from a sentence.

[0104] Optionally, a sparse word vector method may be used to encode the M original sentences and the N translated sentences.

[0105] Optionally, an encoder in an existing neural network translation or classification model may be used to perform matrix operations on an input sentence and output a sentence vector.

[0106] S203: Segment the target parallel corpus according to the obtained vectors to obtain a plurality of bilingual translation segments.

[0107] S204: Perform an alignment operation on each bilingual translation segment to obtain an alignment result of the target parallel corpus.

[0108] The embodiment of the present application first performs sentence segmentation and encoding operations on the target parallel corpus, so that accurate similarity calculation can be performed for each short sentence, sentence pairs that can be used to segment the document are found, the entire document is segmented to obtain multiple short documents, and the alignment result of the long document is obtained by splicing and calculating the similarity of the short documents. Since the embodiment of the present application can segment a long document into short documents and divide the alignment of complex long documents into the splicing and alignment of short documents, it solves the problem that the existing sentence vector technology can only align single sentences to single sentences, or the dynamic programming algorithm cannot accurately process multiple sentences to multiple sentences, thereby improving the accuracy and efficiency of the alignment of the target parallel corpus.

[0109] Optionally, the embodiment of the present application can segment the target parallel corpus according to the first similarity of M*N sentence pairs consisting of any original sentence and any translated sentence in M ​​original sentences and N translated sentences. Correspondingly, Figure 3 A flowchart of another parallel corpus processing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:

[0110] S301: Perform sentence segmentation operation on the target parallel corpus to obtain M sentences of original text of the original document and N sentences of translated text of the translated document in the target parallel corpus.

[0111] S302: Encode the M original sentences and the N translated sentences to obtain a vector corresponding to each original sentence and a vector corresponding to each translated sentence.

[0112] Among them, steps S301 and S302 are implemented in the same way as the above steps S201 and S202, and will not be repeated here.

[0113] S303: Calculate the first similarity of M*N sentence pairs consisting of any original sentence in the M original sentences and any translated sentence in the N translated sentences according to the obtained vectors.

[0114] S304: Determine sentence pairs for segmenting the document according to the first preset rule and the first similarity.

[0115] Optionally, determining sentence pairs for segmenting the document according to the first preset rule and the first similarity includes:

[0116] Sentence pairs whose first similarity is lower than a first preset similarity threshold are deleted from the M*N sentence pairs, and the remaining sentence pairs in the M*N sentence pairs are screened according to a first preset rule; and sentence pairs for segmenting the document are determined according to the screening result.

[0117] Here, the embodiment of the present application deletes sentence pairs whose first similarity is lower than the first preset similarity threshold. At the same time, according to the first preset rule, sentence pairs suitable as positioning points can be screened out, the segmentation points for segmenting the documents can be accurately determined, and the target parallel corpus can be accurately and finely divided, thereby further improving the accuracy of the alignment of the target parallel corpus.

[0118] Optionally, determining sentence pairs for segmenting the document according to the first preset rule and the first similarity includes:

[0119] The first similarities are sorted from high to low, and if the first similarity of the sentence pair is before a preset order among all the first similarities, the sentence pair is determined to be a sentence pair for segmenting the document.

[0120] Optionally, determining sentence pairs for segmenting the document according to the first preset rule and the first similarity includes:

[0121] Deleting sentence pairs whose first similarity is lower than a first preset similarity threshold from the M*N sentence pairs, and screening the remaining sentence pairs in the M*N sentence pairs according to a first preset rule;

[0122] Based on the screening results, sentence pairs for segmenting the document are determined.

[0123] S305: Segmenting the target parallel corpus according to the sentence pairs of the segmented documents to obtain a plurality of bilingual translation segments.

[0124] Optionally, the first similarity may be calculated by replacing a BLEU score with a bilingual evaluation score.

[0125] Optionally, the first similarity may be calculated by calculating the cosine similarity score between any two sentences of the original text and the translated text. Optionally, the first similarity may be obtained by normalizing the cosine similarity between the two sentence vectors, and the calculation formula is as follows:

[0126]

[0127] In the above formula, score(x, y) is the relative similarity of two sentences (here used as an example for the first similarity calculation), cos(x, y) is the cosine similarity score of the vectors obtained by the encoder of the two sentences, x represents the encoded vector of a sentence in language 1, and y represents the encoded vector of a sentence in language 2. The meaning is that for x, find the vector z of the first k sentences closest to it from language 2, then calculate the cosine similarity respectively, and then find the average of the k similarities, and the formula The meaning of is the same. For y, find the first k vectors z that are closest to y from another language 1, then calculate the cosine similarity between y and these k vectors respectively, and then average their similarities.

[0128] Exemplary, Figure 4 A schematic diagram of a segmentation process provided in an embodiment of the present application, such as Figure 4 As shown, after the document 1 is divided into 10 sentences, the document 2 is divided into 13 sentences, Srci and Tgtj (i∈[1,10], j∈[1,13]) represent the i-th and j-th sentences in document 1 and document 2, respectively. Assuming that after calculating the relative similarity and filtering according to the first preset rule, Src4, Tgt3 and Src8, Tgt10 are selected as segmentation points, the bilingual document can be divided into three mutually aligned segments, wherein the bilingual document 1 is divided into three segments Src1-Src4, Src4-Src8, Src9-Src10, and the three segments of the bilingual document 2 are Tgt1-Tgt3, Tgt4-Tgt9, Tgt10-Tgt13, so that the next step of alignment between single parallel paragraphs can be carried out.

[0129] The embodiment of the present application calculates the similarity between all original texts and all translations to determine the sentence pairs, i.e., segmentation points, for segmenting the document, and segments the target mutual translation document according to the sentence pairs of the segmented document. Here, by calculating the similarity between all original texts and all translations, the original texts and translations with the highest similarity can be accurately determined, thereby accurately dividing the target mutual translation segments, further improving the accuracy of the target parallel corpus alignment.

[0130] Optionally, in order to ensure the accuracy of the segmentation results and the determination of the segmentation points, the embodiment of the present application may pre-screen the sentence pairs.

[0131] Optionally, before filtering the remaining sentence pairs in the M*N sentence pairs, the following steps are further included:

[0132] Determine the relationship between the sentences in the remaining sentence pairs; if it is determined based on the relationship that the similarity of any sentence in the to-be-processed sentence pair in the remaining sentence pairs is lower than the similarity of the same sentence in other sentence pairs, delete the to-be-processed sentence pair.

[0133] Here, before screening the remaining sentence pairs in the M*N sentence pairs, the embodiment of the present application screens the sentence pairs to determine the relationship between the sentences in the remaining sentence pairs, so as to facilitate the determination of the sentence pairs used to segment the document in the remaining sentence pairs. In addition, the remaining sentence pairs can be arranged from high to low according to similarity. If any sentence in a sentence pair appears in a sentence pair with a higher score than it, then it can be determined that another sentence has a higher match degree than the current sentence pair, and then the current sentence pair is deleted.

[0134] Optionally, when calculating similarity after combination, all sentences in the original text and the translation are calculated for similarity. That is, if there are N sentences in language 1 and M sentences in language 2, a total of N*M similarities will be obtained. These N*M similarity scores are then sorted from high to low, and the previously matched alignment results are removed from high to low. For example, if the score (the first sentence in language 1, the second sentence in language 2) > the score (the first sentence in language 1, the first sentence in language 2), the alignment result (the first sentence in language 1, the first sentence in language 2) will be removed, because one sentence cannot be aligned to two sentences at the same time, and only one sentence can be selected.

[0135] Optionally, before filtering the remaining sentence pairs in the M*N sentence pairs, the following steps are further included:

[0136] If the ratio of the number of words to the number of characters in a sentence pair exceeds a first preset ratio threshold, the sentence pair is deleted; or, if the ratio of the length of the segments after segmentation according to the labels of a sentence pair exceeds a second preset ratio threshold, the sentence pair is deleted; or, after splicing any sentence in a sentence pair with its previous or next sentence, recalculate the similarity score of the modified sentence pair, and if the similarity score of the new sentence pair becomes higher, delete the sentence pair; or, if the GLEU score of a sentence pair after translation into the same language is lower than the preset score threshold, delete the sentence pair; or, if the number of intersections generated by a sentence pair and other sentence pairs is greater than the preset number of intersections, delete the sentence pair.

[0137] Here, before screening the remaining sentence pairs in the M*N sentence pairs, the embodiment of the present application provides several methods for screening sentence pairs. The sentence pairs are selected based on the ratio of the number of words or characters in the sentence pairs, the ratio of the length of the segments after segmentation by labels, the similarity score, the GLEU score and the number of intersections between the original text and the translation. Sentence pairs that are not suitable as the target parallel corpus for separation are pre-screened and deleted, thereby reducing the workload of subsequently determining the sentence pairs for segmenting the document and further improving the efficiency and accuracy of segmentation and alignment.

[0138] S306: Perform an alignment operation on each bilingual translation segment to obtain an alignment result of the target parallel corpus.

[0139] Optionally, the method for aligning the bilingual translation segments to obtain the concatenated sentences is as follows:

[0140] According to the order of the target parallel corpus, multiple bilingual translation segments are spliced ​​into multiple sentence combinations. Multiple sentence combinations are encoded to obtain a vector corresponding to each combination. According to the obtained vectors, the second similarity between the original text and the translation in the multiple sentence combinations is calculated. According to the second preset rule and the second similarity, the alignment result of the target parallel corpus is obtained.

[0141] It can be understood that when the above-mentioned bilingual translation segments are combined, they can be combined sentence by sentence, which is equivalent to no combination, or they can be combined two sentences by two sentences, three sentences by three sentences, four sentences by four sentences, etc. The specific number of combinations is not specifically limited in this application. By way of example, an upper limit of the number of sentences in a combination can be set, for example, 4 sentences for English and 6 sentences for Chinese and Japanese. This application does not make any specific restrictions on this upper limit, and it can be adjusted according to actual conditions.

[0142] Optionally, the second similarity may be calculated by replacing the BLEU score with a bilingual evaluation score.

[0143] Optionally, the second similarity may be calculated by calculating a cosine similarity normalized score between any sentence combinations of the original text and the translated text, using the same formula as above:

[0144]

[0145] Optionally, the second preset rule here may be to determine a preset second similarity threshold, and determine a sentence combination whose second similarity is greater than the second similarity threshold as the alignment result, or, the second preset rule here may be to sort the second similarities, and determine a certain percentage of sentence combinations with the highest similarity as the alignment result.

[0146] Exemplary, Figure 5 A schematic diagram of an alignment process provided in an embodiment of the present application is shown as follows: Figure 5 As shown, assuming that file 1 has a total of m = 4 sentences and file 2 has a total of n = 3 sentences, after the sentences in file 1 are combined in sequence at most x = 4 sentences and the sentences in file 2 are combined in sequence at most y = 3 sentences, file 1 can obtain S1 = m + (m-1) + (m-2) + ... + (mx) = 10 sentences, and file 2 can obtain S2 = n + (n-1) + (n-2) + ... + (ny) = 6 sentences, as shown in FIG. Figure 5 As shown in the upper right corner, the relative similarity score Score ([Srci, Srci+i, Srci+2, ..., Srci+k], [Tgtj, Tgtj+1, ..., Tgtj+o]) between any two bilingual combinations in the paragraph is then calculated, where i is the i-th sentence in file 1, j is the j-th sentence in file 2, k and o are integers that do not exceed the respective combination sentence limits x and y, and there are a total of S1×S2=10×6=60 scores. The scores are sorted from high to low, and a combination is deleted if any sentence in the combination is included in a combination with a higher score. After eliminating the combination with a score lower than a certain threshold score, the result obtained is the final alignment result, that is, the first sentence in file 1 is aligned with the first and second sentences in file 2, and the third and fourth sentences in file 1 are aligned with the third sentence in file 2, and the alignment result is obtained.

[0147] Here, the embodiment of the present application splices the split short documents according to the original order of the target parallel corpus to obtain multiple sentence combinations, and then calculates the similarity of all the spliced ​​sentence combinations to obtain the sentence combination with the highest similarity between the original text and the translation, thereby achieving accurate alignment and further improving the accuracy of the alignment of the target parallel corpus.

[0148] Figure 6 A schematic diagram of the structure of a parallel corpus processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device of the embodiment of the present application includes: a sentence module 601, an encoding module 602, a segmentation module 603 and an alignment module 604. It should be noted here that the division of the first acquisition module 601, the second sending module 602 and the processing module 603 is only a division of logical functions, and the two can be physically integrated or independent. The parallel corpus processing device here can be the above-mentioned processor 102 itself, or a chip or integrated circuit that realizes the function of the processor 102. It should be noted here that the division of the sentence module 601, the encoding module 602, the segmentation module 603 and the alignment module 604 is only a division of logical functions, and the two can be physically integrated or independent.

[0149] The sentence segmentation module 601 is used to perform a sentence segmentation operation on the target parallel corpus to obtain M original sentences of the original document and N translated sentences of the translated document in the target parallel corpus;

[0150] The encoding module 602 is used to encode the M original sentences and the N translated sentences to obtain a vector corresponding to each original sentence and a vector corresponding to each translated sentence;

[0151] Segmentation module 603, used to segment the target parallel corpus according to the obtained vectors to obtain multiple bilingual translation segments;

[0152] The alignment module 604 is used to perform an alignment operation on each bilingual translation segment to obtain an alignment result of the target parallel corpus.

[0153] Optionally, the segmentation module 603 is specifically used for:

[0154] According to the obtained vectors, the first similarities of M*N sentence pairs consisting of any original sentence in the M original sentences and any translated sentence in the N translated sentences are calculated.

[0155] According to the first preset rule and the first similarity, sentence pairs for segmenting the document are determined.

[0156] According to the sentence pairs of the segmented documents, the target parallel corpus is segmented to obtain multiple bilingual translation segments.

[0157] Optionally, the alignment module 604 is specifically used for:

[0158] According to the order of the target parallel corpus, multiple bilingual translation segments are spliced ​​into multiple sentence combinations.

[0159] Encode multiple sentence combinations and obtain a vector corresponding to each combination.

[0160] According to the obtained vectors, the second similarity between the original text and the translation in the plurality of sentence combinations is calculated.

[0161] According to the second preset rule and the second similarity, an alignment result of the target parallel corpus is obtained.

[0162] Optionally, the segmentation module 603 is specifically used for:

[0163] Sentence pairs whose first similarity is lower than a first preset similarity threshold are deleted from the M*N sentence pairs, and the remaining sentence pairs in the M*N sentence pairs are screened according to a first preset rule; and sentence pairs for segmenting the document are determined according to the screening result.

[0164] Optionally, before screening the remaining sentence pairs in the M*N sentence pairs, the segmentation module 603 is further used to:

[0165] Relationships between sentences in the remaining sentence pairs are determined.

[0166] If it is determined based on the relationship that the similarity of any sentence in the to-be-processed sentence pair in the remaining sentence pairs is lower than the similarity of the same sentence in other sentence pairs, the to-be-processed sentence pair is deleted.

[0167] Optionally, before screening the remaining sentence pairs in the M*N sentence pairs, the segmentation module 603 is further used to:

[0168] If the ratio of the number of words to the number of characters in a sentence pair exceeds a first preset ratio threshold, the sentence pair is deleted; or, if the ratio of the length of the segments after segmentation of a sentence pair according to the labels exceeds a second preset ratio threshold, the sentence pair is deleted; or, after splicing any sentence in a sentence pair with its previous or next sentence in the original text, the similarity of the new sentence pair is calculated, and if the similarity score of the new sentence pair becomes higher, the sentence pair is deleted; or, if the natural language understanding benchmark GLEU score of the sentence pair after translation into the same language is lower than the first preset score, the sentence pair is deleted; or, if the number of intersections generated by a sentence pair and other sentence pairs is greater than the preset number of intersections, the sentence pair is deleted.

[0169] Figure 7 A schematic diagram of a parallel corpus processing device provided for an embodiment of the present application. The components, their connections and relationships, and their functions shown herein are merely examples and do not limit the implementation of the present application described and / or required herein.

[0170] like Figure 7 As shown, the parallel corpus processing device includes: a processor 701 and a memory 702, each of which is connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor 701 can process instructions executed in the parallel corpus processing device, including instructions for graphic information stored in or on the memory for display on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Figure 7 A processor 701 is taken as an example.

[0171] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the method of the parallel corpus processing device in the embodiment of the present application (for example, the attached Figure 6 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the method of implementing the parallel corpus processing device in the above method embodiment.

[0172] The parallel corpus processing device may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0173] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the parallel corpus processing device, such as a touch screen, a keypad, a mouse, or multiple mouse buttons, a trackball, a joystick and other input devices. The output device 704 can be an output device such as a display device of the parallel corpus processing device. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device can be a touch screen.

[0174] The parallel corpus processing device of the embodiment of the present application can be used to execute the technical solutions in the above-mentioned method embodiments of the present application. Its implementation principles and technical effects are similar and will not be repeated here.

[0175] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement any of the above-mentioned parallel corpus processing methods.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

Claims

1. A parallel corpus processing method, characterized in that: include: Performing a sentence segmentation operation on the target parallel corpus to obtain M original sentences of the original document and N translated sentences of the translated document in the target parallel corpus; Encoding the M original texts and the N translated texts to obtain a vector corresponding to each original text and a vector corresponding to each translated text; According to the obtained vectors, the first similarity of M*N sentence pairs consisting of any original sentence in the M original sentences and any translated sentence in the N translated sentences is calculated; Deleting sentence pairs whose first similarity is lower than a first preset similarity threshold from the M*N sentence pairs, and screening the remaining sentence pairs in the M*N sentence pairs according to the first preset rule; According to the screening results, K sentence pairs for segmenting the document are determined, where K is an integer greater than or equal to 1; The original document is divided into K+1 parts by taking the K first sentences belonging to the original document in the sentence pairs of the segmented document as the dividing points, and the translated document is divided into K+1 parts by taking the K second sentences belonging to the translated document in the sentence pairs of the segmented document as the dividing points, to obtain a plurality of bilingual inter-translation segments; the K+1 parts into which the original document is divided correspond to the K+1 parts into which the translated document is divided, and one part of the original document and one part corresponding to the translated document constitute one bilingual inter-translation segment; For each bilingual translation segment: according to the order of the target parallel corpus, the sentences corresponding to the original text segments in the bilingual translation segment are combined to obtain S1 first sentence combinations, and the sentences corresponding to the target text segments in the bilingual translation segment are combined to obtain S2 second sentence combinations; both S1 and S2 are integers greater than or equal to 1; Encode each sentence combination in the S1 first sentence combinations and the S2 second sentence combinations to obtain a vector corresponding to each sentence combination; Calculate, according to the vectors corresponding to each sentence combination, a second similarity of S1*S2 sentence combinations consisting of any one sentence combination in the S1 first sentence combinations and any one sentence combination in the S2 second sentence combinations; According to the second preset rule and the second similarity, an alignment result of the target parallel corpus is obtained.

2. The method according to claim 1, characterized in that Before screening the remaining sentence pairs in the M*N sentence pairs, the method further includes: determining relations between sentences in the remaining sentence pairs; If it is determined based on the relationship that the similarity of any sentence in the to-be-processed sentence pair in the remaining sentence pairs is lower than the similarity of the same sentence in other sentence pairs, the to-be-processed sentence pair is deleted.

3. The method according to claim 1 or 2, characterized in that: The first preset rule includes: If the ratio of the number of words to the number of characters in a sentence pair exceeds a first preset ratio threshold, the sentence pair is deleted; or, If the ratio of the lengths of the segments of a sentence pair after being segmented according to the labels exceeds a second preset ratio threshold, the sentence pair is deleted; or, Then delete the sentence pair, concatenate any sentence in a sentence pair with its previous sentence or next sentence, and recalculate the similarity score of the modified sentence pair. If the similarity score of the new sentence pair becomes higher, delete the sentence pair; or, If the GLEU score of a sentence pair after being translated into the same language is lower than the preset score threshold, the sentence pair will be deleted; or, If the number of intersections between a sentence pair and other sentence pairs is greater than the preset number of intersections, the sentence pair is deleted.

4. A parallel corpus processing device, characterized in that: include: A sentence segmentation module, used for performing a sentence segmentation operation on the target parallel corpus to obtain M original sentences of the original document and N translated sentences of the translated document in the target parallel corpus; An encoding module, used for encoding the M original texts and the N translated texts to obtain a vector corresponding to each original text and a vector corresponding to each translated text; A segmentation module, used for calculating, based on the obtained vectors, a first similarity of M*N sentence pairs consisting of any original sentence in the M original sentences and any translated sentence in the N translated sentences; Deleting sentence pairs whose first similarity is lower than a first preset similarity threshold from the M*N sentence pairs, and screening the remaining sentence pairs in the M*N sentence pairs according to the first preset rule; According to the screening results, K sentence pairs for segmenting the document are determined, where K is an integer greater than or equal to 1; The original document is divided into K+1 parts by taking the K first sentences belonging to the original document in the sentence pairs of the segmented document as the dividing points, and the translated document is divided into K+1 parts by taking the K second sentences belonging to the translated document in the sentence pairs of the segmented document as the dividing points, to obtain a plurality of bilingual inter-translation segments; the K+1 parts into which the original document is divided correspond to the K+1 parts into which the translated document is divided, and one part of the original document and one part corresponding to the translated document constitute one bilingual inter-translation segment; An alignment module is used for: for each bilingual translation segment: according to the order of the target parallel corpus, the sentences corresponding to the original text segments in the bilingual translation segment are combined to obtain S1 first sentence combinations, and the sentences corresponding to the target text segments in the bilingual translation segment are combined to obtain S2 second sentence combinations; both S1 and S2 are integers greater than or equal to 1; Encode each sentence combination in the S1 first sentence combinations and the S2 second sentence combinations to obtain a vector corresponding to each sentence combination; Calculate, according to the vectors corresponding to each sentence combination, a second similarity of S1*S2 sentence combinations consisting of any one sentence combination in the S1 first sentence combinations and any one sentence combination in the S2 second sentence combinations; According to the second preset rule and the second similarity, an alignment result of the target parallel corpus is obtained.

5. A parallel corpus processing device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the parallel corpus processing method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the parallel corpus processing method according to any one of claims 1 to 3 is implemented.

Images