Text alignment method and system based on dynamic programming

Abstract

The present invention provides a text alignment method and system based on dynamic programming, including: calculating the initial similarity between the vector corresponding to any original sentence and the vector corresponding to each translated sentence in the target translated text; obtaining the initial similarity between any original sentence and the first K translated sentences of any translated text by forward search, obtaining the initial similarity between any translated text and the first K original text sentences of any original text by backward search, obtaining the candidate translated sentences corresponding to each original text sentence with optimized similarity; deleting the wrong anchor edge; treating the remaining unaligned original text sentences as original text preset windows again, treating the remaining unaligned translated text sentences as translated text alignment windows again, repeating the above process, and obtaining the optimal translated text sentence corresponding to each original text sentence. The embodiment of the present invention utilizes the original translated text sentence sequence relationship to optimize the alignment effect, and can also complete the splicing of multiple translated texts corresponding to one original text, and has a high fault tolerance mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text alignment method and system based on dynamic programming. Background Art

[0002] Text alignment is to divide the document into sentences according to language rules based on the existence of both the original text and the translation, and then find the corresponding translation sentence for each original sentence to align the sentences.

[0003] The traditional alignment system based on statistical rules extracts feature information based on the most original statistical information of the language, such as keywords, word edit distance or number of words. Finally, based on the extracted features, the similarity between the original sentence and the translated sentence is calculated to determine whether it is a bilingual parallel sentence. Different languages ​​need to be customized to adapt to the corresponding rules. Except for the TF-IDF based on document word frequency, which can adapt to multiple languages, other language features are either difficult to customize due to language characteristics, or the corpus is not rich enough, and the formulated rules are not universal, resulting in uneven traditional alignment effects.

[0004] In recent years, language processing technology based on neural networks has developed rapidly. Based on massive network text data, it is easy to obtain a large amount of data in different languages, and provide a unified technology for feature processing for multiple languages, that is, embedding technology.

[0005] In terms of form, embedding technology is to use a low-dimensional dense vector to represent an object (Object), where the object can be a word, a sentence, or a document. Sentences in different languages ​​can be converted into vectors in the same vector space through embedding technology, and operations between vectors can include semantic relationships between words and sentences. This provides a consistent technical foundation for the processing of universal text alignment rules in multiple languages.

[0006] Although text alignment using embedding technology can completely rely on sentence similarity to achieve sentence alignment, it still faces the problem of choosing a similarity threshold and the constraint of document word order. Conventional sentence alignment does not make corresponding optimization processing for document alignment. For example, if the similarity threshold is too high, there will be very few aligned sentences. Repairing the alignment relationship of the remaining sentences on this basis will be inaccurate, and the semantic similarity relationship provided by the alignment vector is not fully explored and utilized, that is, there are too few sentence alignment anchor points; if the similarity threshold is too low, although the sentence alignment anchor points are dense, there are anchor point errors. In the text alignment task, if the alignment anchor points are misplaced, the subsequent sentences will completely miss the correct alignment window. Summary of the invention

[0007] The present invention provides a text alignment method and system based on dynamic programming, which are used to solve the defect of low alignment accuracy in the prior art and realize high-precision text alignment.

[0008] The present invention provides a text alignment method based on dynamic programming, comprising:

[0009] Obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text;

[0010] For any original text sentence in the target original text that is located within the original text preset window, calculating the initial similarity between a vector corresponding to the any original text sentence and a vector corresponding to each translated text sentence in the target translated text that is located within the translated text preset window;

[0011] For any translation sentence in the translation preset window, obtain the initial similarity between any original sentence and the first K translation sentences of any translation sentence by forward search, obtain the initial similarity between any translation sentence and the first K original sentences of any original sentence by backward search, obtain the optimized similarity between each original sentence and each translation sentence, K>0, and K is a positive integer, the first K translation sentences are the first K translation sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translation sentence;

[0012] Obtaining candidate translation sentences corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0013] Obtaining the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, deleting the wrong anchor edges in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edges represent the alignment relationship between the original sentences and the candidate translated sentences, and the wrong anchor edges are the anchor edges whose optimized similarity ranks last in a preset proportion among all anchor edges;

[0014] The remaining unaligned original sentences are re-used as the original preset windows, and the remaining unaligned translated sentences are re-used as the translated alignment windows. The above process is repeated. If each original sentence finds a corresponding candidate translated sentence, the optimal translated sentence corresponding to each original sentence is obtained.

[0015] A text alignment method based on dynamic programming provided by the present invention also includes:

[0016] If the preset number of cycles is reached, for the remaining unaligned translation sentences, the remaining unaligned translation sentences are spliced ​​with the target anchor translation sentence, and if the optimized similarity between the target anchor original sentence and the spliced ​​translation sentence is greater than the optimized similarity between the target anchor original sentence and the target anchor translation sentence, the spliced ​​translation sentence is re-used as the target anchor translation sentence, and the target anchor translation sentence is the aligned translation sentence above the remaining unaligned translation sentences.

[0017] A text alignment method based on dynamic programming provided by the present invention also includes:

[0018] If the optimized similarity between the target anchor original sentence and the concatenated translation sentence is not greater than the optimized similarity between the target anchor original sentence and the target anchor translation sentence, no processing is performed.

[0019] According to a text alignment method based on dynamic programming provided by the present invention, the calculating of the initial similarity between the vector corresponding to any original sentence and the vector corresponding to each translated sentence in the target translated text within the preset window includes:

[0020] The cosine similarity between the vector corresponding to any original sentence and the vector corresponding to each translated sentence in the target translated text within the preset window is calculated, which is the initial similarity.

[0021] According to a text alignment method based on dynamic programming provided by the present invention, the calculating of the initial similarity between the vector corresponding to any original sentence and the vector corresponding to each translated sentence in the target translated text within the preset window includes:

[0022] The L2 distance between the vector corresponding to any original sentence and the vector corresponding to each translated sentence in the target translated text within the preset window is calculated, which is the initial similarity.

[0023] According to a text alignment method based on dynamic programming provided by the present invention, before obtaining the optimized similarity between any original sentence and any translated sentence according to the initial similarity between the first K translated sentences of any original sentence and the initial similarity between any translated sentence and the last K original sentences of any original sentence, the method further includes:

[0024] Regularization is performed on the initial similarity to obtain the regularized initial similarity, and the regularized initial similarity is used as the initial similarity again.

[0025] According to a text alignment method based on dynamic programming provided by the present invention, before obtaining a candidate translation sentence corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence, the method further includes:

[0026] Regularization is performed on the optimized similarity to obtain the regularized optimized similarity, and the regularized optimized similarity is used as the optimized similarity again.

[0027] The present invention also provides a text alignment system based on dynamic programming, comprising:

[0028] A vector module is used to obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text;

[0029] An initial similarity module, for calculating, for any original text sentence in the target original text located within the original text preset window, an initial similarity between a vector corresponding to the any original text sentence and a vector corresponding to each translated text sentence in the target translated text located within the translated text preset window;

[0030] A similarity optimization module is used for obtaining, for any translated sentence in the preset window, an initial similarity between any original sentence and the first K translated sentences of any translated sentence by forward search, obtaining an initial similarity between any translated sentence and the first K original sentences of any original sentence by backward search, obtaining an optimized similarity between each original sentence and each translated sentence, K>0, and K is a positive integer, the first K translated sentences are the first K translated sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translated sentence;

[0031] A candidate module, for obtaining a candidate translation sentence corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0032] An optimization module, configured to obtain a first longest increasing subsequence corresponding to all original sentences and a second longest increasing subsequence corresponding to all translated sentences, and delete an erroneous anchor edge in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edge represents an alignment relationship between the original sentence and the candidate translated sentence, and the erroneous anchor edge is an anchor edge whose optimized similarity ranks last in a preset proportion among all anchor edges;

[0033] The iterative output module is used to treat the remaining unaligned original sentences as the original preset window, and the remaining unaligned translation sentences as the translation alignment window, and repeat the above process. If each original sentence finds a corresponding candidate translation sentence, the optimal translation sentence corresponding to each original sentence is obtained.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned text alignment methods based on dynamic programming are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned text alignment methods based on dynamic programming are implemented.

[0036] The present invention provides a text alignment method and system based on dynamic programming, which utilizes the document sentence sequence relationship and the dynamic programming algorithm to adaptively solve the sentence alignment misalignment chain reaction caused by inaccurate key sentences and anchor points. The dynamic programming algorithm is completely adaptive and can complete the extraction of key anchor sentences and sentence splicing alignment without pre-setting a threshold. That is, the original translation sentence sequence relationship is utilized, and the alignment effect is optimized. At the same time, the splicing of multiple translation sentences corresponding to one original sentence can be completed, and a high fault tolerance mechanism is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A flowchart of a text alignment method based on dynamic programming provided by the present invention;

[0039] Figure 2 A structural schematic diagram of a text alignment system based on dynamic programming provided by the present invention;

[0040] Figure 3 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The embodiment of the present invention provides a text alignment method based on dynamic programming, such as Figure 1 As shown, the method includes:

[0043] In the embodiment of the present invention, alignment is used to indicate the process of matching mutually translated segments between texts in different languages, and is also often used to indicate the final matching result obtained in the process.

[0044] 110, obtaining a vector corresponding to each original sentence in the target original text and a vector corresponding to each translated sentence in the target translated text;

[0045] The target original text is split into multiple sentences, called original sentences. The existing multilingual text embedding (Sentence Embedding) technology is used to calculate the vector corresponding to each original sentence. The original sentence is converted to the same dimensional space through the vector. The vector can be regarded as the distributed expression of the sentence.

[0046] In the embodiment of the present invention, text embedding technology includes but is not limited to LASER, BERT pre-training model and other technologies. Any embedding technology that can convert original sentences into the same dimension and can express the relationship model between sentences through vectors can be applied to this method.

[0047] 120, for any original text sentence in the target original text that is located within the original text preset window, calculating the initial similarity between a vector corresponding to the any original text sentence and a vector corresponding to each translated text sentence in the target translated text that is located within the translated text preset window;

[0048] Next, the sentence similarity is calculated. In the embodiment of the present invention, a source text preset window and a translation preset window are first set. The lengths of the two windows can be determined according to actual needs. The source text preset window and the translation preset window are to ensure that the amount of calculation for each iteration is not too large, thereby controlling the amount of calculation for the entire solution.

[0049] For any original sentence in the original text preset window, the similarity between the original sentence and each translated sentence in the translated text preset window is calculated.

[0050] In a specific embodiment, the specific method for calculating the similarity between the original sentence and the translated sentence is: calculating the similarity between the vector corresponding to the original sentence and the vector corresponding to each translated sentence. The similarity at this time is called the initial similarity.

[0051] In one embodiment, the initial similarity is the cosine similarity between the two vectors.

[0052] In another embodiment, the initial similarity is the L2 distance between the two vectors.

[0053] In another preferred embodiment, the initial similarity needs to be regularized to improve the accuracy of the vector similarity. The purpose of the regularization is to unify all the initial similarities into the same dimension, and its effect is similar to normalization.

[0054] According to the same steps as above, the initial similarity between each original sentence in the preset original window and each translated sentence in the preset translated window is calculated.

[0055] Traditional embedding technology does not fully utilize the sequential relationship of the document sentence sequence, that is, the context of the original sentence must be consistent with the context of the translated sentence.

[0056] In the embodiment of the present invention, by utilizing the word order relationship of the document, not only the alignment effect can be optimized, but also excellent fault tolerance can be achieved.

[0057] 130, for any translated sentence in the preset window, obtain the initial similarity between any original sentence and the first K translated sentences of any translated sentence by forward search, obtain the initial similarity between any translated sentence and the first K original sentences of any original sentence by backward search, obtain the optimized similarity between each original sentence and each translated sentence, K>0, and K is a positive integer, the first K translated sentences are the first K translated sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translated sentence;

[0058] Since the corpora used for embedding pre-training of the target original language and the target translation language may be inconsistent, for example, the pre-training corpus of the Chinese version of BERT is much smaller than the pre-training corpus of the English version. This is because the current Internet is expected to be dominated by English materials, and multi-language model training cannot guarantee the absolute consistency of the corpus. In some embodiments, certain regularization measures are still needed to deal with this situation.

[0059] In order to better represent the similarity between the original sentence and the translated sentence, based on the initial similarity calculated previously, the optimized similarity between the original sentence and the translated sentence is calculated in combination with the sentence context. The specific steps are as follows:

[0060] Taking any original sentence in the preset original window and any translated sentence in the preset translated window as an example, the initial similarity between the original sentence and each translated sentence in the preset translated window is obtained and sorted, and the top K translated sentences with the largest initial similarity are found through forward search, and the translated sentences with the initial similarity in the top K are taken as the top K translated sentences of the translated sentence.

[0061] In addition, the initial similarity between each original sentence and the translation sentence in the preset original window is used and sorted, and the first K original sentences with the largest initial similarity are found through backward search, and the first K original sentences are used as the first K original sentences of the original sentence.

[0062] In the embodiment of the present invention, the average of the initial similarity between the original sentence and the first K translated sentences of the translated sentence and the initial similarity between the translated sentence and the first K original sentences of the original sentence is used as the optimized similarity between the original sentence and the translated sentence.

[0063] The optimized similarity between each original sentence and each translated sentence is calculated in the same way.

[0064] 140, obtaining a candidate translation sentence corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0065] In the embodiment of the present invention, the highest optimization similarity indicates that the possibility of alignment between two sentences is the greatest, and the translation sentence with the highest optimization similarity is used as the candidate translation sentence.

[0066] For example, for a target original sentence 1, search for the first K translation sentences with the highest similarity in the target translation. When K=3, take out the first three translation sentences with the highest similarity, for example, the translation sentences {1,3,5} form a corresponding relationship, original sentence {1}-->translation sentence {1,3,5}; then search for the first three sentences with the highest similarity in the target original sentence from the translation sentence 1 of the target translation, for example, translation {1}-->original {1,2,4}. It should be noted that the directions of these two operations are different. One is the original {1}-->translation {1,3,5}, this is a forward search, and the other is translation {1}-->original {1,2,4}, this is a backward search. The same thing is that both search for the top K sentences with the highest similarity as candidate relationships. The difference is that one searches from original to translated, and the other searches from translated to original. The candidate sentences obtained by these two searches are different, but both are retained as candidate connections for the next step, because the final result only requires one sentence {1} in the original to correspond to one sentence {1} in the translated version. All redundant relationships will be filtered out in the layers of optimization and screening.

[0067] 150, obtaining the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, and deleting the wrong anchor edge in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edge represents the alignment relationship between the original sentence and the candidate translated sentence, and the wrong anchor edge is the anchor edge whose optimized similarity ranks last in a preset proportion among all anchor edges;

[0068] In order to further optimize the above alignment results, since in the first alignment, it is inevitable that there will be noise, alignment errors and other operations, the sentence alignment relationship must include not being arranged according to the target original text and the target translation sentence sequence, because there may be a sentence of the target original text that is incorrectly matched to another sentence of the target translation, which will interfere with the entire alignment matching list. Such a correspondence should be discarded in the process of anchor point selection. Therefore, in the embodiment of the present invention, it is necessary to first find the wrong alignment relationship and then delete the wrong alignment relationship.

[0069] The alignment relationship between the original sentence and the candidate translation sentence is called an anchor edge, the original sentence of the anchor edge is called the original anchor, and the translation sentence of the anchor edge is called the translation anchor.

[0070] First, the longest increasing subsequence corresponding to the target text is calculated, which is called the first longest increasing subsequence. The longest increasing subsequence refers to finding a subsequence in a given numerical sequence, so that the values ​​of the elements of this subsequence increase successively, and the length of this subsequence is as large as possible. The elements in the longest increasing subsequence are not necessarily continuous in the target text.

[0071] Similarly, find the longest increasing subsequence corresponding to the target translation, which is called the second longest increasing subsequence.

[0072] According to the optimized similarity of each anchor edge, the anchor edges are sorted and the anchor edges with the lowest similarity are deleted. That is to say, the anchor edge with the last similarity indicates that the correlation between the two is not strong. Therefore, the alignment relationship with low similarity is deleted. That is to say, the deleted original sentence is regarded as the remaining unaligned original sentence, and the deleted translated sentence is regarded as the remaining unaligned translated sentence.

[0073] 160, the remaining unaligned original sentences are used as the original preset window, and the remaining unaligned translation sentences are used as the translation alignment window, and the above process is repeated. If each original sentence finds a corresponding candidate translation sentence, the optimal translation sentence corresponding to each original sentence is obtained.

[0074] Then, the remaining unaligned original sentences are used as the original preset window, and the remaining unaligned translated sentences are used as the translated preset window. Steps 120 to 150 are repeated to perform loop iterations. When each original sentence finds a corresponding candidate translated sentence, the corresponding candidate translated sentence is used as the best translated sentence.

[0075] The embodiment of the present invention provides a text alignment method based on dynamic programming, which utilizes the document sentence sequence relationship and the dynamic programming algorithm to adaptively solve the sentence alignment dislocation chain reaction caused by inaccurate key sentences and anchor points. The dynamic programming algorithm is completely adaptive and can complete the extraction of key anchor sentences and sentence splicing alignment without pre-setting a threshold. That is, the original translation sentence sequence relationship is utilized, and the alignment effect is optimized. At the same time, the splicing of multiple translation sentences corresponding to one original sentence can be completed, and a high fault tolerance mechanism is provided.

[0076] Based on the above embodiment, preferably, it further includes:

[0077] If the preset number of cycles is reached, for the remaining unaligned translation sentences, the remaining unaligned translation sentences are spliced ​​with the target anchor translation sentence, and if the optimized similarity between the target anchor original sentence and the spliced ​​translation sentence is greater than the optimized similarity between the target anchor original sentence and the target anchor translation sentence, the spliced ​​translation sentence is re-used as the target anchor translation sentence, and the target anchor translation sentence is the aligned translation sentence above the remaining unaligned translation sentences.

[0078] Based on the above embodiment, preferably, it further includes:

[0079] If the optimized similarity between the target anchor original sentence and the concatenated translation sentence is not greater than the optimized similarity between the target anchor original sentence and the target anchor translation sentence, no processing is performed.

[0080] Automatically merge the remaining unaligned sentences, because there may be a sentence in the original text that will be translated into multiple sentences in the target text. In this way, when calculating the embedding similarity, it will cause insufficient similarity and it will never be restored.

[0081] The conventional method is to use a dynamic programming algorithm similar to the edit distance to concatenate multiple sentences of the translation into one sentence, and then calculate the similarity. This is prone to many-to-many problems, such as when there are three 1-1 sentences, a 3-3 alignment result is generated, because the traditional technology uses the method of first merging and then finding the alignment anchor point.

[0082] This problem is well solved here. In the embodiment of the present invention, the method of first finding the alignment anchor point and then fusing is adopted. Because the splicing measures are only taken for the context above and below the key sentence, and the self-increasing and decreasing conditions of the sentence similarity are required to adaptively select the splicing, there will be no tendency to many-to-many situation.

[0083] Here, due to the existence of the anchor point of the key sentence, these scattered sentences must be at the top of the window that needs to be aligned. Therefore, you only need to try to splice the part of the translated sentence below the sentence anchor point with the anchor sentence and recalculate the similarity. If a higher similarity is obtained, the spliced ​​sentence can be set as valid. If the similarity is reduced, it means that the sentence below is not an extra sentence in the translation process and no processing is done.

[0084] The embodiment of the present invention provides a text alignment system based on dynamic programming, such as Figure 2 As shown, the system includes a vector module 201, an initial similarity module 202, an optimized similarity module 203, a candidate module 204, an optimization module 205 and an iterative output module 206, wherein:

[0085] The vector module 201 is used to obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text;

[0086] The initial similarity module 202 is used to calculate the initial similarity between the vector corresponding to any original sentence in the target original text located within the original text preset window and the vector corresponding to each translation sentence in the target translation located within the translation preset window;

[0087] The similarity optimization module 203 is used for obtaining, for any translated sentence in the preset window, an initial similarity between any original sentence and the first K translated sentences of any translated sentence by forward search, obtaining an initial similarity between any translated sentence and the first K original sentences of any original sentence by backward search, obtaining an optimized similarity between each original sentence and each translated sentence, K>0, and K is a positive integer, the first K translated sentences are the first K translated sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translated sentence;

[0088] The candidate module 204 is used to obtain a candidate translation sentence corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0089] The optimization module 205 is used to obtain the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, and delete the wrong anchor edge in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edge represents the alignment relationship between the original sentence and the candidate translated sentence, and the wrong anchor edge is the anchor edge whose optimization similarity ranks last in a preset proportion among all anchor edges;

[0090] The iterative output module 206 is used to treat the remaining unaligned original sentences as the original preset window, and the remaining unaligned translation sentences as the translation alignment window, and repeat the above process. If each original sentence finds a corresponding candidate translation sentence, the optimal translation sentence corresponding to each original sentence is obtained.

[0091] This embodiment is a system embodiment corresponding to the above method. For details, please refer to the above method embodiment. This system embodiment will not be described in detail here.

[0092] An embodiment of the present invention provides an electronic device, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a text alignment method based on dynamic programming, the method comprising:

[0093] Obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text;

[0094] For any original text sentence in the target original text that is located within the original text preset window, calculating the initial similarity between a vector corresponding to the any original text sentence and a vector corresponding to each translated text sentence in the target translated text that is located within the translated text preset window;

[0095] For any translation sentence in the translation preset window, obtain the initial similarity between any original sentence and the first K translation sentences of any translation sentence by forward search, obtain the initial similarity between any translation sentence and the first K original sentences of any original sentence by backward search, obtain the optimized similarity between each original sentence and each translation sentence, K>0, and K is a positive integer, the first K translation sentences are the first K translation sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translation sentence;

[0096] Obtaining candidate translation sentences corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0097] Obtaining the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, deleting the wrong anchor edges in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edges represent the alignment relationship between the original sentences and the candidate translated sentences, and the wrong anchor edges are the anchor edges whose optimized similarity ranks last in a preset proportion among all anchor edges;

[0098] The remaining unaligned original sentences are re-used as the original preset windows, and the remaining unaligned translated sentences are re-used as the translated alignment windows. The above process is repeated. If each original sentence finds a corresponding candidate translated sentence, the optimal translated sentence corresponding to each original sentence is obtained.

[0099] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0100] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute a text alignment method based on dynamic programming provided by the above methods, the method comprising:

[0101] Obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text;

[0102] For any original text sentence in the target original text that is located within the original text preset window, calculating the initial similarity between a vector corresponding to the any original text sentence and a vector corresponding to each translated text sentence in the target translated text that is located within the translated text preset window;

[0103] For any translation sentence in the translation preset window, obtain the initial similarity between any original sentence and the first K translation sentences of any translation sentence by forward search, obtain the initial similarity between any translation sentence and the first K original sentences of any original sentence by backward search, obtain the optimized similarity between each original sentence and each translation sentence, K>0, and K is a positive integer, the first K translation sentences are the first K translation sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translation sentence;

[0104] Obtaining candidate translation sentences corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0105] Obtaining the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, deleting the wrong anchor edges in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edges represent the alignment relationship between the original sentences and the candidate translated sentences, and the wrong anchor edges are the anchor edges whose optimized similarity ranks last in a preset proportion among all anchor edges;

[0106] The remaining unaligned original sentences are re-used as the original preset windows, and the remaining unaligned translated sentences are re-used as the translated alignment windows. The above process is repeated. If each original sentence finds a corresponding candidate translated sentence, the optimal translated sentence corresponding to each original sentence is obtained.

[0107] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned text alignment method based on dynamic programming, the method comprising:

[0108] Obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text;

[0109] For any original text sentence in the target original text that is located within the original text preset window, calculating the initial similarity between a vector corresponding to the any original text sentence and a vector corresponding to each translated text sentence in the target translated text that is located within the translated text preset window;

[0110] For any translation sentence in the translation preset window, obtain the initial similarity between any original sentence and the first K translation sentences of any translation sentence by forward search, obtain the initial similarity between any translation sentence and the first K original sentences of any original sentence by backward search, obtain the optimized similarity between each original sentence and each translation sentence, K>0, and K is a positive integer, the first K translation sentences are the first K translation sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translation sentence;

[0111] Obtaining candidate translation sentences corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence;

[0112] Obtaining the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, deleting the wrong anchor edges in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edges represent the alignment relationship between the original sentences and the candidate translated sentences, and the wrong anchor edges are the anchor edges whose optimized similarity ranks last in a preset proportion among all anchor edges;

[0113] The remaining unaligned original sentences are used as the original preset window, and the remaining unaligned translation sentences are used as the translation alignment window. The above process is repeated. If each original sentence finds a corresponding candidate translation sentence, the optimal translation sentence corresponding to each original sentence is obtained.

[0114] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text alignment method based on dynamic programming, characterized in that: include: Obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text; For any original sentence in the target original text that is within the original text preset window, calculating the initial similarity between the vector corresponding to the any original sentence and the vector corresponding to each translated sentence in the target translated text that is within the translated text preset window; For any translation sentence in the translation preset window, an initial similarity between any original sentence and the first K translation sentences of any translation sentence is obtained by forward search, and an initial similarity between any translation sentence and the first K original sentences of any original sentence is obtained by backward search, and an optimized similarity between any original sentence and any translation sentence is obtained according to the initial similarity between any original sentence and the first K translation sentences of any translation sentence and the initial similarity between any translation sentence and the first K original sentences of any original sentence, K>0, and K is a positive integer, the first K translation sentences are the first K translation sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translation sentence; Obtaining candidate translation sentences corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence; Obtaining the first longest increasing subsequence corresponding to all original sentences and the second longest increasing subsequence corresponding to all translated sentences, deleting the wrong anchor edges in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edges represent the alignment relationship between the original sentences and the candidate translated sentences, and the wrong anchor edges are the anchor edges whose optimized similarity ranks last in a preset proportion among all anchor edges; The remaining unaligned original sentences are re-used as the original preset windows, and the remaining unaligned translation sentences are re-used as translation alignment windows. The above process is repeated. If each original sentence finds a corresponding candidate translation sentence, the optimal translation sentence corresponding to each original sentence is obtained.

2. The text alignment method based on dynamic programming according to claim 1, characterized in that: Also includes: If the preset number of cycles is reached, for the remaining unaligned translation sentences, the remaining unaligned translation sentences are spliced ​​with the target anchor translation sentence, and if the optimized similarity between the target anchor original sentence and the spliced ​​translation sentence is greater than the optimized similarity between the target anchor original sentence and the target anchor translation sentence, the spliced ​​translation sentence is re-used as the target anchor translation sentence, and the target anchor translation sentence is the aligned translation sentence above the remaining unaligned translation sentences.

3. The text alignment method based on dynamic programming according to claim 2, characterized in that: Also includes: If the optimized similarity between the target anchor original sentence and the concatenated translation sentence is not greater than the optimized similarity between the target anchor original sentence and the target anchor translation sentence, no processing is performed.

4. The text alignment method based on dynamic programming according to any one of claims 1 to 3, characterized in that: The calculating of the initial similarity between the vector corresponding to any one of the original sentences and the vector corresponding to each translated sentence in the target translated text within the translated text preset window includes: The cosine similarity between the vector corresponding to any original sentence and the vector corresponding to each translation sentence in the target translation within the translation preset window is calculated, which is the initial similarity.

5. The text alignment method based on dynamic programming according to any one of claims 1 to 3, characterized in that: The calculating of the initial similarity between the vector corresponding to any one of the original sentences and the vector corresponding to each translated sentence in the target translated text within the translated text preset window includes: The L2 distance between the vector corresponding to any original sentence and the vector corresponding to each translation sentence in the target translation within the translation preset window is calculated, which is the initial similarity.

6. The text alignment method based on dynamic programming according to any one of claims 1 to 3, characterized in that: Before obtaining the optimized similarity between any original sentence and any translated sentence according to the initial similarity between any original sentence and the first K translated sentences of any translated sentence and the initial similarity between any translated sentence and the first K original sentences of any original sentence, the method further includes: Regularization is performed on the initial similarity to obtain the regularized initial similarity, and the regularized initial similarity is used as the initial similarity again.

7. The text alignment method based on dynamic programming according to any one of claims 1 to 3, characterized in that: Before obtaining the candidate translation sentence corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence, the method further includes: Regularization is performed on the optimized similarity to obtain the regularized optimized similarity, and the regularized optimized similarity is used as the optimized similarity again.

8. A text alignment system based on dynamic programming, characterized in that: include: A vector module is used to obtain the vector corresponding to each original sentence in the target original text and the vector corresponding to each translated sentence in the target translated text; An initial similarity module, for calculating, for any original text sentence in the target original text that is within the original text preset window, an initial similarity between a vector corresponding to any original text sentence and a vector corresponding to each translated text sentence in the target translated text that is within the translated text preset window; A similarity optimization module is used for obtaining, for any translation sentence in the translation preset window, an initial similarity between any original sentence and the first K translation sentences of any translation sentence by forward search, obtaining an initial similarity between any translation sentence and the first K original sentences of any original sentence by backward search, and obtaining an optimized similarity between any original sentence and any translation sentence according to the initial similarity between any original sentence and the first K translation sentences of any translation sentence and the initial similarity between any translation sentence and the first K original sentences of any original sentence, K>0, and K is a positive integer, the first K translation sentences are the first K translation sentences with the highest similarity to any original sentence, and the first K original sentences are the first K original sentences with the highest similarity to any translation sentence; A candidate module, for obtaining a candidate translation sentence corresponding to each original sentence according to the optimized similarity between each original sentence and each translation sentence; An optimization module, configured to obtain a first longest increasing subsequence corresponding to all original sentences and a second longest increasing subsequence corresponding to all translated sentences, and delete an erroneous anchor edge in the first longest increasing subsequence and the second longest increasing subsequence, wherein the anchor edge represents an alignment relationship between the original sentence and the candidate translated sentence, and the erroneous anchor edge is an anchor edge whose optimized similarity ranks last in a preset proportion among all anchor edges; The iterative output module is used to treat the remaining unaligned original sentences as the original preset window, and the remaining unaligned translation sentences as the translation alignment window, and repeat the above process. If each original sentence finds a corresponding candidate translation sentence, the optimal translation sentence corresponding to each original sentence is obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the text alignment method based on dynamic programming as claimed in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the text alignment method based on dynamic programming as claimed in any one of claims 1 to 7 are implemented.

Images