Professional terminology vocabulary alignment replacement method based on Transformer translation model

Abstract

The invention discloses a method for aligning and replacing professional terminology vocabulary based on a translation model of a Transformer, constructs and trains a translation model based on a Transformer, inputs a source language text to be translated into a trained translation model, translates to obtain an initial translation of a target language, obtains multiple reference correlation matrices of source language words and target language words at the same time, aligns the source language words and target language words according to the reference correlation matrices, obtains an aligned word pair, searches for source language terms in a pre-set professional terminology library in a source language sentence, and if so, queries a word set aligned with the source language term in the initial translation, replaces the target language words aligned with the source language term in the initial translation with the professional terminology translation, thereby obtaining a final translation. The invention utilizes the correlation matrix in the translation model to realize the alignment of the source language and the translation, and utilizes the professional terminology library to correct the translation, thereby improving the accuracy of the translation.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and more specifically, relates to a professional terminology vocabulary alignment replacement method based on a Transformer translation model. Background Art

[0002] Machine translation is the process of converting a source language into another target language through a computer. For example, the translation software that is often used in daily life can translate English into Chinese, where English is a source language and Chinese is the target language. Although manual translation is highly accurate, it is time-consuming and laborious. Although the translation of machine translation is not as accurate as manual translation, its speed is much faster than manual translation. Therefore, when a large amount of text needs to be translated and the accuracy requirement is not so high, such as browsing tasks of massive data, the advantages of machine translation are reflected. For those things that cannot be completed by manual translation, machine translation may only take a few hours or even minutes to complete. Translation plays an important role for humans. On the one hand, due to the differences in language, culture and geographical location, translation has become an important demand; on the other hand, translation has also accelerated the integration of different civilizations and promoted the development of the world. Precisely because the demand for translation is so huge, machine translation has always been one of the most meaningful research topics.

[0003] Nowadays, under some conditions, the results of machine translation are close to those of manual translation. However, in some specific fields, such as the medical field, the results are still unsatisfactory. Because the medical field has a lot of specialized terms, these specialized terms are not common in the existing public parallel corpus datasets and are difficult to be learned by the model. In some specific scenarios, the translation accuracy of these professional terms is very high, so how to use the terminology library to correct and improve the translation obtained by the translation model is an important problem that needs to be solved to adapt the translation model to some specific fields. Traditional alignment replacement algorithms based on attention mechanisms often only rely on the cross-attention score calculated by the last layer of the model decoder. Therefore, the alignment results obtained are not necessarily accurate. Other alignment replacement algorithms add constraints during the model decoding process, which sometimes affect the final translation results of the model.

[0004] In summary, the current method of allowing the model to fully utilize the terminology library to correct and improve the translation has the problem that the alignment results are not accurate enough and may affect the quality of the translation generated by the model, thereby affecting the adaptation of the translation model to specific fields. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a professional terminology vocabulary alignment and replacement method based on a Transformer translation model, which utilizes the correlation matrix in the translation model to achieve accurate alignment of the input source language text with the translation generated by the translation model, and uses a professional terminology library to correct the translation, thereby improving the accuracy of the translation.

[0006] In order to achieve the above-mentioned object of the invention, the professional terminology vocabulary alignment replacement method based on the Transformer translation model of the present invention comprises the following steps:

[0007] S1: Build a Transformer-based translation model, including M encoding layers and M decoding layers, and use the pre-collected parallel corpora of the source language and the target language to train the translation model;

[0008] S2: Input the source language text to be translated into the Transformer-based translation model trained in step S1, and translate it to obtain the initial translation in the target language. The number of words in the source language text is recorded as D r , the number of target language words in the initial translation is D t ; When the initial translation is obtained, the size D calculated by the cross attention mechanism in the M decoding layers in the translation model is obtained t ×D r A correlation matrix, each element represents the correlation value between the source language word and the target language word at the corresponding position; according to the needs, N decoding layers among the M decoding layers are selected as reference decoding layers, and the corresponding correlation matrix is ​​used as the reference correlation matrix R n , n=1,2,…,N;

[0009] S3: For each target language word in the initial translation, determine its N reference relevance matrices R n The source language word in the source language text with the largest correlation value is taken as the pending source language word W d,n , and its corresponding correlation value is recorded as C d,n , d=1,2,…,D t ; The N pending source language words W corresponding to each target language word d,n The set of undetermined source language words that constitute the target language word φ d , remember the set of words in the undetermined source language φ d The number of undetermined source language words is K. For the kth undetermined source language word, its frequency f is counted k,n and the mean of the correlation values Weighted score of the undetermined source language word α and β represent the preset weights and α+β=1; finally, among the K pending source language words of each target language word, select the word with score S k,n The largest pending source language word is used as the aligned source language word of the target language word, thereby obtaining an aligned word pair;

[0010] S4: According to the pre-set professional terminology databases of the source language and the target language, query whether each source language term exists in the input source language text. If not, no operation is performed. If so, the alignment relationship between the source language word and the target language word in the initial translation determined in step S3 is used to find the target language word set aligned with the source language term in the initial translation, and replace the target language word aligned with the source language term in the initial translation with the professional term target language translation corresponding to the source language term, so as to obtain the final translation.

[0011] The invention discloses a professional terminology vocabulary alignment and replacement method based on a Transformer-based translation model, which constructs and trains a Transformer-based translation model, inputs a source language text to be translated into a trained translation model, obtains an initial translation in a target language through translation, obtains a plurality of reference correlation matrices of source language words and target language words, aligns the source language words and the target language words according to the reference correlation matrices to obtain aligned word pairs, searches for source language terms in a pre-set professional terminology library in a source language sentence, and if so, queries a word set aligned with the source language term in the initial translation, replaces the target language words aligned with the source language term in the initial translation with the professional terminology translation, and obtains a final translation.

[0012] The present invention has the following beneficial effects:

[0013] 1) The present invention draws on the idea of ​​model integration and no longer uses only the correlation matrix generated by the last layer of the decoder in the translation model, but instead uses multiple layers of correlation matrices to achieve word alignment, thereby enhancing the accuracy and robustness of word alignment;

[0014] 2) In the present invention, the alignment and replacement of professional terms are separated from the generation of translation model translations. That is, some words in the translations are replaced and corrected only after the translation model generates the translations. This ensures that the translation model is not interfered by the outside world during the decoding process and does not affect the translation performance of the model itself.

[0015] 3) The present invention can also introduce a heuristic method to determine whether to replace professional terms and filter out some erroneous alignment results, thereby further ensuring the translation quality of the final translation.

[0016] In summary, the present invention can fully utilize the properties of the Transformer-based translation model without affecting the performance of the translation model itself, accurately align the input source language text and the translation generated by the model, and then use the professional terminology library to correct the translation of some professional terms in the translation, thereby improving the translation accuracy of the translation model for texts in specific fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a specific implementation method of the professional terminology vocabulary alignment replacement method based on the Transformer translation model of the present invention;

[0018] Figure 2 It is a structural diagram of the Transformer-based translation model in this embodiment. DETAILED DESCRIPTION

[0019] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0020] Example

[0021] Figure 1 FIG. 1 is a flowchart of a specific implementation method of the professional terminology vocabulary alignment replacement method based on the Transformer translation model of the present invention. Figure 1 As shown, the specific steps of the professional terminology vocabulary alignment replacement method based on the Transformer translation model of the present invention include:

[0022] S101: Build and train a Transformer-based translation model:

[0023] Transformer is a structure proposed in the paper "Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need[J]. arXiv, 2017." This structure abandons the common RNN, LSTM and other structures, and only uses the Attention mechanism to improve the processing ability of long sentences, so it is widely used in the field of NLP, such as machine translation, question-answering system, text summarization and speech recognition. Therefore, the present invention constructs a translation model based on Transformer, including M encoding layers and M decoding layers, and uses pre-collected parallel corpora of the source language and the target language to train the translation model.

[0024] Figure 2is a structural diagram of the Transformer-based translation model in this embodiment. Figure 2 As shown, the Transformer-based translation model in this embodiment adopts 6 encoding layers (encoder) and 6 decoding layers (decoder). Each layer in the encoder encoder contains two modules, a self-attention module and a feedforward neural network module. The self-attention mechanism can help the current node pay attention to the global information of the input sentence, thereby obtaining the semantics of the context. Each layer of the decoder decoder also contains the two modules mentioned in the encoding layer, but there is also a cross-attention mechanism cross-attention module between the two modules. This module can help the current node obtain the key content that needs to be paid attention to in the source language sentence. Therefore, the correlation matrix calculated by cross-attention contains the alignment information between the source language sentence and the model predicted translation words.

[0025] S102: Translate to obtain the translation and obtain the reference correlation matrix:

[0026] The source language text to be translated is input into the Transformer-based translation model trained in step S101, and the initial translation in the target language is obtained by translation. The number of words in the source language text is recorded as D r , the number of target language words in the initial translation is D t . When the initial translation is obtained, the size D calculated by the cross attention mechanism in the M decoding layers in the translation model is obtained. t ×D r A correlation matrix, where each element represents the correlation value between the source language word and the target language word at the corresponding position. Select N decoding layers from the M decoding layers as reference decoding layers as needed, and the corresponding correlation matrix is ​​used as the reference correlation matrix R n , n=1,2,…,N.

[0027] It can be seen that the present invention uses the correlation matrix of multiple decoding layers instead of relying solely on the correlation matrix of the last layer for alignment. Because in the training process, the data in the training set often contains noise, which causes occasional deviations in the calculation of the correlation matrix. The alignment result obtained by using only the correlation matrix of the last layer has poor robustness and may have obvious errors in some cases. By using the correlation matrix of multiple layers of reference decoders, the influence of certain noise can be eliminated and the robustness of the alignment algorithm can be enhanced.

[0028] Since the functions that can be realized by each decoding layer of the translation model are different, the correlation matrices corresponding to them are also different. The present invention finds that it is not possible to obtain a more accurate alignment result by simply fusing the correlation matrices of all decoding layers. Therefore, in order to make the selected correlation matrix more accurate, the present invention sets up an experiment to find the optimal combination of decoding layer correlation matrices. The specific method is as follows:

[0029] Use an external alignment tool (such as GIZA++, etc.) to obtain the alignment results of the source language and the target language as their true labels, then input the source language and the target language into the translation model, obtain the correlation matrix calculated by the cross-attention mechanism in the M decoding layers of the translation model, perform alignment extraction based on each correlation matrix, compare the extracted alignment results with the true labels, and calculate the alignment error rate (Alignment Error Rate, AER) of the alignment results. The decoding layer corresponding to the correlation matrix with an alignment error rate less than a preset threshold is used as the reference decoding layer.

[0030] by Figure 2 Taking the translation model shown in FIG. 1 as an example, it is found through experiments that the alignment error rate generated by the last three layers of the six decoding layers is lower than the preset threshold, while the first three layers are shallow layers of the decoder and may have different tasks to be completed from the last three layers, so the alignment results generated are quite different from those of the last three layers, resulting in a higher alignment error rate. Therefore, in this embodiment, only the correlation matrix of the last three layers is used as a reference for alignment.

[0031] S103: Perform word alignment based on the reference correlation matrix:

[0032] Due to the principle of the attention mechanism, during the translation process, the correlation matrix generated by the trained translation model naturally reflects the contribution of the representation of each word in the source language text to the generation of each word in the translation. The higher the contribution ratio, the greater the corresponding correlation value in the correlation matrix, and the greater the probability that the source language word and the target language word are aligned. Therefore, the present invention determines the alignment result based on the N reference correlation matrices in step S102, and the specific method is as follows:

[0033] For each target language word in the initial translation, determine its N reference relevance matrices R n The source language word in the source language text with the largest correlation value is taken as the pending source language word W d,n , and its corresponding correlation value is recorded as C d,n , d=1,2,…,D t . The N pending source language words W corresponding to each target language word d,n The set of undetermined source language words that constitute the target language word φd , remember the set of words in the undetermined source language φ d The number of undetermined source language words is K. For the kth undetermined source language word, its frequency f is counted k,n (i.e., among N pending source language words W d,n The ratio of the number of occurrences of and N) and the mean of the correlation values Weighted score of the undetermined source language word α and β represent the preset weights and α+β=1. Finally, among the K undetermined source language words of each target language word, select the word with score S k,n The largest pending source language word is used as the aligned source language word of the target language word, thereby obtaining an aligned word pair.

[0034] In this embodiment, the source language is set to English, the target language is set to Chinese, the source language text is "COVID-19is aninfectious disease", and the translation is "Pneumonia is an infectious disease". As mentioned above, in this embodiment, the correlation matrix obtained by the last three decoding layers is used as the reference correlation matrix. Table 1 is the first reference correlation matrix in this embodiment.

[0035] COVID-19 is an infectious disease pneumonia 0.63 0.12 0.08 0.04 0.13 yes 0.21 0.45 0.12 0.06 0.23 A sort of 0.11 0.18 0.28 0.22 0.21 infectious disease 0.02 0.22 0.07 0.33 0.36

[0036] Table 1

[0037] Table 2 is the second reference correlation matrix in this embodiment.

[0038] COVID-19 is an infectious disease pneumonia 0.33 0.13 0.08 0.34 0.12 yes 0.24 0.35 0.18 0.16 0.07 A sort of 0.16 0.11 0.35 0.22 0.21 infectious disease 0.03 0.12 0.17 0.32 0.36

[0039] Table 2

[0040] Table 3 is the third reference correlation matrix in this embodiment.

[0041] COVID-19 is an infectious disease pneumonia 0.46 0.18 0.09 0.13 0.14 yes 0.26 0.35 0.13 0.14 0.12 A sort of 0.20 0.33 0.24 0.10 0.13 infectious disease 0.17 0.14 0.01 0.37 0.31

[0042] Table 3

[0043] First, the source language word with the greatest correlation in each reference correlation matrix is ​​selected as the pending source language word. Table 4 is a list of pending target language words and their scores for each source language word in this embodiment.

[0044] <![CDATA[R1]]> <![CDATA[R2]]> <![CDATA[R3]]> pneumonia 0(0.63) 3(0.34) 0(0.46) yes 1(0.45) 1(0.35) 1(0.35) A sort of 2(0.28) 2(0.35) 1(0.33) infectious disease 4(0.36) 4(0.36) 3(0.37)

[0045] Table 4

[0046] In Table 4, column numbers are used to replace source language words, and the corresponding correlation values ​​are in brackets. Taking the target language word "pneumonia" as an example, its pending source language words include: "COVID-19", with an occurrence frequency of 2 / 3 and a correlation mean of 0.545; "infectious", with an occurrence frequency of 1 / 3 and a correlation mean of 0.34. In this embodiment, the weights α and β are both set to 0.5, and the scores corresponding to the two pending source language words are 0.61 and 0.34 respectively, so the alignment result is "COVID-19". Similarly, the alignment result of "is", the alignment result of "a kind" is "an", and the alignment result of "infectious" is "infectious".

[0047] S104: Professional term search and replacement:

[0048] According to the pre-set professional terminology libraries of the source language and the target language, it is queried whether each source language term exists in the input source language text. If not, no operation is performed. If it exists, the alignment relationship between the source language word and the target language word in the initial translation determined in step S103 is used to find the target language word set aligned with the source language term in the initial translation, and the target language word aligned with the source language term in the initial translation is replaced with the professional term target language translation corresponding to the source language term, so as to obtain the final translation.

[0049] A source language term may contain multiple words. For a source language term in a source language sentence, it may be aligned to multiple words in the initial translation. Therefore, according to this situation, a heuristic method can be used to determine whether to perform a replacement operation on the source language term in the source language text. The specific method is as follows:

[0050] The number of words contained in the source language term to be replaced is recorded as L1, and the number of words in the initial translation that are aligned with the source language term to be replaced is recorded as L2. If v×L1<L2<u×L1, the replacement operation is performed, that is, the part corresponding to the source language term in the initial translation is replaced with the corresponding term translation in the term library, otherwise the replacement operation is not performed, where v and u are coefficients preset according to the translation habits between the source language and the target language.

[0051] By using the above method, it can be ensured that the final translation will not be affected by these incorrect alignment results.

[0052] Table 5 is an example of the term library in this embodiment.

[0053] COVID-19 Etoposide cisplatin mometasone SARS-CoV-2 Coronavirus disease Etoposide Cisplatin Mometasone Novel Coronavirus

[0054] Table 5

[0055] In this embodiment, it is first found that only the term "COVID-19" exists in the sentence of the source language, and its alignment result is ["COVID-19":"pneumonia"].

[0056] For the heuristic algorithm, the coefficients u=2 and v=0.5 are set, and for the alignment result of the term "COVID-19" ["COVID-19": "pneumonia"], the length of the source language term L1=1, and the length of the corresponding word set "pneumonia" L2=1, which does not exceed the set range, so the replacement operation can be performed for the term "COVID-19".

[0057] Therefore, "pneumonia" is replaced with the term translation "COVID-19" in the term library. Therefore, the final translation of the source language text "COVID-19is an infectious disease" is "COVID-19 is an infectious disease". It can be seen that the translation processed by the present invention is more accurate than the initial translation.

[0058] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A professional terminology vocabulary alignment replacement method based on Transformer translation model, characterized in that The following steps are involved: S1: Build a Transformer-based translation model, including M encoding layers and M decoding layers, and use the pre-collected parallel corpora of the source language and the target language to train the translation model; S2: Input the source language text to be translated into the Transformer-based translation model trained in step S1, and translate it to obtain the initial translation in the target language. The number of words in the source language text is recorded as D r , the number of target language words in the initial translation is D t ; When the initial translation is obtained, the size D calculated by the cross attention mechanism in the M decoding layers in the translation model is obtained t ×D r A correlation matrix, each element represents the correlation value between the source language word and the target language word at the corresponding position; According to the need, select N decoding layers from the M decoding layers as reference decoding layers, and the corresponding correlation matrix as the reference correlation matrix R n , n=1,2,…,N; S3: For each target language word in the initial translation, determine its N reference relevance matrices R n The source language word in the source language text with the largest correlation value is taken as the pending source language word W d,n , and its corresponding correlation value is recorded as C d,n , d=1,2,…,D t ; The N pending source language words W corresponding to each target language word d,n The set of undetermined source language words that constitute the target language word φ d , remember the set of words in the undetermined source language φ d The number of undetermined source language words is K. For the kth undetermined source language word, its frequency f is counted k,n and the mean of the correlation values Weighted score of the undetermined source language word α and β represent the preset weights and α+β=1; finally, among the K pending source language words of each target language word, select the word with score S k,n The largest pending source language word is used as the aligned source language word of the target language word, thereby obtaining an aligned word pair; S4: According to the pre-set professional terminology databases of the source language and the target language, query whether each source language term exists in the input source language text. If not, no operation is performed. If so, the alignment relationship between the source language word and the target language word in the initial translation determined in step S3 is used to find the target language word set aligned with the source language term in the initial translation, and replace the target language word aligned with the source language term in the initial translation with the professional term target language translation corresponding to the source language term, so as to obtain the final translation.

2. The method for aligning and replacing professional terminology according to claim 1, characterized in that: The method for selecting the reference decoding layer in step S2 is as follows: Use an external alignment tool to obtain the alignment results of the source language and the target language as their true labels, then input the source language and the target language into the translation model, obtain the correlation matrix calculated by the cross-attention mechanism in the M decoding layers of the translation model, perform alignment extraction based on each correlation matrix, compare the extracted alignment results with the true labels, calculate the alignment error rate of the alignment results, and use the decoding layer corresponding to the correlation matrix with an alignment error rate less than a preset threshold as the reference decoding layer.

3. The method for aligning and replacing professional terminology according to claim 1, characterized in that: The step S4 also includes using a heuristic method to determine whether to perform a replacement operation on the source language term existing in the source language text, and the specific method is as follows: The number of words contained in the source language term to be replaced is recorded as L1, and the number of words in the initial translation that are aligned with the source language term to be replaced is recorded as L2. If v×L1<L2<u×L1, the replacement operation is performed, otherwise the replacement operation is not performed, where v and u are coefficients preset according to the translation habits between the source language and the target language.

Images