A medical machine translation method based on reinforcement learning

Abstract

The present invention discloses a medical machine translation method based on reinforcement learning, comprising the steps of constructing a parallel corpus, establishing a machine translation model, pre-training the basic parameters of the machine translation model, constructing a test set to calculate the translation quality score, constructing a reinforcement learning training set, and fine-tuning the optimized parameters of the machine translation model. This invention makes the translation of professional content in the medical vertical field more accurate and convenient, helping medical professionals, research institutes, institutions, and pharmaceutical and medical device companies to quickly understand and learn about the world's most advanced medical information knowledge and medical industry development trends, effectively improving the clinical diagnosis and treatment level and consultation efficiency of medical practitioners.

Description

Technical Field

[0001] The present invention relates to a medical machine translation method, in particular to a medical machine translation method based on reinforcement learning, and belongs to the technical field of machine translation. Background Art

[0002] From a global perspective, the global pharmaceutical and medical service markets are now more closely connected than ever before. China's growing global consumption of medical services and medical products and the increasing requirements for clinical trials in China have led to an increasing demand for medical translation.

[0003] Due to the seriousness and specificity of medicine, medical translation is a highly specialized field. In principle, it is crucial not only to ensure general accuracy, coherence, logical connections, and inter-lexical connections, but also to ensure the accuracy of medical terminology and concepts, as well as conformity to clinical or scientific reading habits. This ensures accurate translation of the target language into the target language, ensuring readers receive a largely intact copy of the original text. Even the slightest discrepancy can have serious consequences.

[0004] Traditional manual medical translation requires the collaboration of multiple parties, including experienced and skilled linguists, industry experts, and project managers. Furthermore, translation requirements vary across specialized fields, such as basic medicine, clinical medicine, and pharmacy. This often requires the assistance of domain experts with extensive experience in specific areas. This also imposes high quality control standards on translation results and is time-consuming and labor-intensive. Machine translation emerged to free humans from the burden of translation tasks.

[0005] The Bilingual Evaluation Understudy (BLEU) is an effective translation performance metric that helps humans quickly evaluate the quality of machine translation. Google has released the Paraphrase Adversaries from Word Scrambling (PAWS) dataset for English and the Paraphrase Adversaries from Word Scrambling extension (PAWS-X) dataset for multiple languages. Both PAWS and PAWS-X consist of highly structured sentence pairs with a high degree of lexical overlap, and are often used to identify whether a pair of sentences have the same meaning. The paper "mT5: A massively multilingual pre-trained text-to-text transformer" proposes a multilingual pre-trained model (MT5) for sentence rewriting, which has achieved good results. These existing technologies have laid the foundation for research on high-quality translation methods. Summary of the Invention

[0006] The purpose of the present invention is to provide a medical machine translation method based on reinforcement learning.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A medical machine translation method based on reinforcement learning, comprising the following steps:

[0009] Step 1: Construct a parallel corpus: The parallel corpus includes a general corpus sub-corpus and a medical corpus sub-corpus. The general corpus sub-corpus and the medical corpus sub-corpus have the same structure and both include one or more parallel corpora. The parallel corpora consist of the first language, the first language text, the second language and the second language text. The length of the parallel corpus meets the preset length range.

[0010] Step 2: Build a machine translation model: The machine translation model includes an encoder and a decoder. The decoder has the same number of layers as the encoder. The output of the encoder is connected to the corresponding input of the decoder.

[0011] Step 3: Pre-training the machine translation model: extracting corpus from the general corpus sub-database and the medical corpus sub-database according to a preset ratio to construct a basic parameter training set, inputting the basic parameter training set into the machine translation model to train the basic parameters of the machine translation model;

[0012] Step 4: Calculate the translation quality score: Extract corpus from the general corpus sub-database and the medical corpus sub-database according to the preset ratio to construct a test set. Input the first language, first language text, and second language of the test set into the machine translation model trained in step 3 to obtain the translated text. Compare the translated text with the second language text and calculate the BLEU score.

[0013] Step 5: Constructing a reinforcement learning training set: Filter the parallel corpora in the test set whose BLEU scores are lower than the preset threshold score and use them as the basic corpora. Process the basic corpora one by one to construct a reinforcement learning training set. This includes the following specific steps:

[0014] Step 5-1: Construct pseudo-parallel corpus: Use the following three methods to construct pseudo-parallel corpus corresponding to the basic prediction:

[0015] Method 1: Divide the first language text in the basic corpus into more than one word, randomly extract n words from them, where n>1, and replace them with corresponding synonyms;

[0016] Method 2: Translate the second language text of the basic corpus into a language different from the first and second languages, and then translate it back to the second language;

[0017] Method 3: Train the mt5 model to obtain a sentence rewriting model; input the basic corpus into the trained sentence rewriting model to obtain a pseudo-parallel corpus;

[0018] Step 5-2: Use cosine similarity to score each pseudo-parallel corpus, remove those with scores below a preset threshold and those that are completely identical to the original text, and add the remaining pseudo-parallel corpora to the reinforcement learning training set;

[0019] Step 6: Fine-tune the machine translation model: Use the reinforcement learning training set to fine-tune the machine translation model for one or more rounds to obtain the optimized parameters of the machine translation model.

[0020] Furthermore, the medical machine translation method based on reinforcement learning further includes a terminology intervention step, including the following specific steps:

[0021] Step A-1: ​​Create a medical terminology glossary with the data format being [English, Chinese, translation direction], where the translation direction is English to Chinese or Chinese to English;

[0022] Step A-2: Searching for medical terms in the text to be translated, replacing the medical terms in the text to be translated with placeholders; and searching the medical term list to obtain corresponding translation terms;

[0023] Step A-3: using the machine translation model to translate the text to be translated processed in step A-2;

[0024] Step A-4: Replace the placeholders in the translation text obtained in step A-3 with the translated words.

[0025] Furthermore, the medical machine translation method based on reinforcement learning further includes a model compression step, including the following specific steps:

[0026] Step A: Generate the predicted probability of each target word through the softmax function:

[0027]

[0028] Where j = 1, 2, ..., K, z is a vector; T is the temperature coefficient, greater than 1;

[0029] Step B: Knowledge distillation training: The trained model is used as the teacher model, and the model with reduced model layers is used as the student model. The true label is the hard label, and the predicted probability value output by the teacher model is the soft label. The cross entropy loss is calculated for the output of the student model and the hard label, and the mean squared error loss is calculated for the soft label. The weighted sum of the cross entropy loss and the mean squared error loss is used as the final loss function.

[0030]

[0031] L=αL ce_loss +(1-Δ)L mse_loss

[0032]

[0033] Where: N represents the number of samples, M represents the number of categories, log is the logarithm of the base e, y ic Indicates whether sample i is category c, y ic Take 1 or 0, p ic It represents the probability that the predicted sample i belongs to category c, and α is the weight.

[0034] Furthermore, the medical machine translation method based on reinforcement learning also includes a pruning step: reducing the number of network layers to a preset number of layers, reducing the accuracy of the parameters to a preset number of bits through quantization, and using step 5 again to optimize the student model parameters.

[0035] Furthermore, the machine translation model includes an input layer, a convolutional neural network, an encoder, and a decoder. The encoder consists of a sequentially cascaded input layer and the first to 12 encoder layers with the same structure. The decoder consists of a sequentially cascaded input layer and the first to 12 decoder layers with the same structure. The convolutional neural network consists of a sequentially cascaded convolutional layer, a pooling layer, and an output layer. The input layer structure of the encoder and decoder is the same, both consisting of an embedding sublayer and a position sublayer.

[0036] Each encoder layer consists of a multi-head self-attention sublayer, a first residual sublayer, a feedforward network sublayer, and a second residual sublayer cascaded in sequence. After word segmentation of the first language text, the first language word sequence [x1, x2, ..., xm] is obtained. The first language word sequence [x1, x2, ..., xm] is input into the embedding sublayer and the position sublayer respectively. The outputs of the embedding sublayer and the position sublayer are added and then input into the multi-head self-attention sublayer and the first residual sublayer of the first encoder layer respectively. The output end of the first residual sublayer of the first encoder layer is connected to the corresponding input end of the second residual sublayer of the first encoder layer, and the output end of the second residual sublayer is used as the output of the first encoder layer. Each decoder layer consists of a multi-head self-attention sublayer, a first residual sublayer, an encoder-decoder attention sublayer, a second residual sublayer, a feedforward network sublayer, and a third residual sublayer cascaded in sequence.

[0037] The output end of the first residual sublayer of the first decoder layer is connected to the corresponding input end of the second residual sublayer of the first decoder layer, the corresponding input end of the second residual sublayer of the first decoder layer is connected to the corresponding input end of the third residual sublayer of the first decoder layer, and the output end of the third residual sublayer of the first decoder layer is used as the output of the first decoder layer;

[0038] The output word encoding information matrix C of the 12th layer encoder is added to the output semantic information matrix T of the convolutional neural network to obtain the vector matrix S:

[0039] S=C+αT

[0040] Where, α is the sentence semantic information weight;

[0041] Input the vector matrix S to the encoder-decoder attention sublayer of each decoder layer;

[0042] The starting character START is input into the embedding sublayer and position sublayer of the decoder, and is processed by the 1st to 12th decoder layers to obtain the predicted value y1. The predicted value y1 is fed back into the embedding sublayer and position sublayer to obtain the predicted word y2. The predicted value yi, i>1, is fed back into the embedding sublayer and position sublayer to obtain the predicted value yi+1, until the predicted value yi+1 is the end symbol END.

[0043] By adopting the above technical solution, the beneficial effects of the present invention are:

[0044] 1. This invention is based on machine translation and performs refined pre-processing of medical corpus, which improves translation quality and meets the special requirements of the medical field for translation quality;

[0045] 2. The present invention uses knowledge distillation, pruning, and quantization techniques to fine-tune the model multiple times, reducing the size of the model without sacrificing accuracy, improving the efficiency of the translation model, and achieving high concurrency and high performance support for the translation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 This is the structural diagram of the machine translation model.

[0048] Figure 2 It is a flowchart of reinforcement learning with pre-training + fine-tuning.

[0049] Figure 3 It is a flowchart of model optimization.

[0050] Example 1:

[0051] Step 1: Build a parallel corpus

[0052] The collected medical data resources were preprocessed to generate parallel corpora in the [English-Chinese] and [Chinese-English] formats. Specific steps included: data filtering (removing control characters, escape characters, and URL symbols); length filtering (removing sentence pairs that were too short or too long); and using the fast_align tool to score the parallel corpora and filter sentence pairs that exceeded a set threshold. This yielded high-quality medical corpus data, for example, [“There are still some alternative strategies, such as anterior chamber paracentesis, ALPI and argon laser pupilloplasty, corneal indentation.”] and [“There are still some alternative strategies, such as anterior chamber paracentesis, ALPI and argon laser pupilloplasty, corneal indentation.”]. To enable simultaneous translation from Chinese to English and from English to Chinese, the input text needs to include a language ID. The data format is [zh “There are still some alternative strategies, such as anterior chamber paracentesis, ALPI and argon laser pupilloplasty, corneal indentation.”, en “There are still some alternative strategies, such as anterior chamber paracentesis, ALPI and argon laser pupilloplasty, corneal indentation.”].

[0053] Step 2: Build a machine translation model

[0054] The structure of the translation model is as follows Figure 1As shown in the figure, the translation model is based on a Transformer architecture with a self-attention mechanism. The overall architecture utilizes an encoder-decoder framework, consisting of 12 encoder layers and 12 decoder layers. A multi-head attention mechanism is employed to capture rich context and hierarchical information, enabling the model to focus on more features and achieving more accurate translation. Each encoder layer consists of a multi-head self-attention and position-based feedforward network layer, while each decoder layer consists of a multi-head self-attention, encoder-decoder attention, and position-based feedforward network layer. In machine translation, vocabulary is highly variable, and word meanings dynamically change with context and sentence meaning. This can lead to complex situations such as polysemy, which can lead to inaccurate translation. Therefore, during decoding, a convolutional neural network is used to extract sentence semantic information. This information is then integrated into the decoder's decoding process, combining word context and sentence semantics. The convolutional neural network consists of an input layer, convolutional layers, pooling layers, and an output layer. During model training, by integrating semantic features from the entire sentence, more source language information is learned. Experimental verification has shown that this can mitigate polysemy and improve translation quality. The specific steps of the model translation are shown below.

[0055] Step 2-1: Input the segmented source language word sequence [x1, x2, ..., xm] into the model's Embedding layer to obtain word embeddings and position embeddings. Add the word embeddings and position embeddings together to obtain the word representation vector matrix.

[0056] Step 2-2: Pass the word representation vector matrix into the Encoder, and perform weighted summation through the self-attention layers of the 12 Encoders to obtain the encoding information matrix C of all words in the sentence.

[0057] Step 2-3: Pass the source language word sequence through the input layer, convolution layer, pooling layer and output layer of CNN to output the semantic information matrix T of the sentence.

[0058] Step 2-4: Align the semantic information matrix T and the word encoding information matrix C.

[0059] Step 2-5: Since the semantic information of a sentence plays an auxiliary role in the translation process, the parameter α is set as the weight of the sentence semantic information, and S = C + αT is used as the vector matrix output by the final encoder. α needs to be learned multiple times in the model to find the optimal value.

[0060] Step 2-6: Combine the vector matrix S and the translation start symbol " <start>"Input 12 Decoder layers to get the predicted value y1.

[0061] Step 2-7: Input the vector matrix S and the currently translated word i into the Encoder-DecoderAttention layer of the Decoder to obtain the attention matrix of the predicted word and the input sentence. Predict the next word i+1 based on the attention matrix and the last decoded embedding.

[0062] Step 2-8: Until the end of the sentence is predicted <end>"Decoding completed.

[0063] Step 2-9: Decoder output output = (batch_size, token_num, vocab_size), which is decoded by the tokenizer to obtain the final prediction result.

[0064] Step 3: Pre-training + fine-tuning the machine translation model

[0065] The process of pre-training + fine-tuning training model is as follows Figure 2 As shown in the figure. In the pre-training phase, the Chinese-English basic model is first trained using all parallel corpora. Then, the model is fine-tuned for several rounds by freezing the number of layers and reducing the learning rate. In each round of fine-tuning, the BLEU score is calculated for the translated text in the parallel corpus. The BLEU calculation formula is:

[0066]

[0067]

[0068]

[0069] Among them, BP is the length penalty factor, c is the length of the machine translation sentence, r is the length of the reference translation sentence, and w n It is 1 / N, the upper limit of N is 4, n_gram represents a set of phrases of length n words, Count clip (n_gram) represents the number of n_grams in the reference, and Count(n_gram′) represents the number of n_grams in the candidate.

[0070] Filter parallel corpora with scores below the preset threshold as the basic corpus, expand the basic corpus, and continue to fine-tune the model with the expanded corpus until these translation issues are alleviated. The expansion steps are as follows:

[0071] Step 3-1: Randomly extract n words from the sentence to be expanded and replace them with random synonyms in the synonym dictionary.

[0072] Step 3-2: Translate the original English monolingual corpus into French, Japanese, and other languages, and then translate the translated text back into English.

[0073] Step 3-3: Use the PAWS-X dataset to train a Chinese-English sentence rewriting model based on the mt5 model for subsequent sentence expansion.

[0074] Step 3-4: Input the data to be expanded into the sentence rewriting model, set different parameters, and obtain rewritten sentences of different degrees.

[0075] Step 3-5: Use cosine similarity to score the pseudo-parallel corpus, remove sentence pairs with too low scores and those that are completely identical to the original text, and finally add them to the expanded corpus.

[0076] Step 4: Terminology Intervention

[0077] In real-world applications, we often encounter proper nouns and industry terms such as drug names, company names, and English abbreviations. For example, the abbreviation "asd" should be correctly translated as "atrial septal defect." Machine translation engines struggle to accurately translate these specialized terms. By adding a terminology intervention feature, we can more accurately translate medical terms using a user glossary.

[0078] Placeholder replacement method:

[0079] Step 4-1: Construct a glossary based on medical vocabulary. The data format is [English, Chinese, translation direction], such as [asd, atrial septal defect, en2zh]

[0080] Step 4-2: First replace the terms in the original text with special symbols, and then submit it to machine translation. For example, "asd" is replaced with ▂ID='1'▂, [Clinical analysis of 61patients of▂ID='1'▂].

[0081] Step 4-3: Replace the special symbols in the returned machine translation results with Chinese terms according to the glossary, for example, [Clinical analysis of 61 patients with atrial septal defect].

[0082] Word alignment method:

[0083] Step 4-1: Construct a glossary based on medical vocabulary. The data format is [English, Chinese, translation direction], such as [asd, atrial septal defect, en2zh]

[0084] Step 4-2: Translate the original text. For example, if the original text is [Clinical analysis of 61 patients of asd], the translation result is [Clinical analysis of 61 patients with asd].

[0085] Step 4-3: After the translation is completed, the attention value of each word is calculated based on the output matrix of the decoder's cross attention layer to obtain the word relevance score.

[0086] Step 4-4: Obtain multiple results based on the multi-head attention mechanism, and set a threshold to filter the final results corresponding to the source and target words.

[0087] Step 4-5: Find the translation corresponding to the term in the original text and replace it with the corresponding Chinese term. For example, find the translation corresponding to "asd" and replace it with "atrial septal defect" to get the final translation result [Clinical analysis of 61 patients with atrial septal defect]

[0088] For specific medical scenarios, specific medical attribute features are introduced into the underlying model, giving the product leading translation accuracy in the medical industry.

[0089] The BLEU score for Chinese-English translation at the sentence level is higher than 48, the accuracy of date translation is greater than 99%, and the proportion of sentences with manual evaluation scores of 4 or above exceeds 85%.

[0090] Step 5: Machine Translation Model Optimization

[0091] Machine translation models are deep learning networks, often over-parameterized. Network parameters are highly redundant, and different parameters contribute significantly to performance. Deploying these models consumes significant resources and places high demands on device storage and computing power. To reduce resource usage, it's necessary to compress the fine-tuned model.

[0092] Compression process such as Figure 3 As shown. First, the knowledge distillation technology is used to compress the model. The principle is that the knowledge learned by the original model as the teacher model guides the training of the student model with smaller parameters, so that the student model can learn the generalization ability of the teacher model and has comparable performance to the teacher model, but with a significantly reduced number of parameters and faster inference speed. The student model still has the structure of the teacher, and the parameters of one of the two layers of the teacher model are selected as the parameters for initialization of the student. The machine translation model generates the predicted probability of each target word through the softmax function. The formula of the softmax function is:

[0093]

[0094] Where j = 1, 2, ..., K, and z is a vector. When performing knowledge distillation, a temperature coefficient T is added to the original softmax function. The value of T is greater than 1 to make the probability distribution smoother and amplify the similarity information between categories. The modified softmax function formula is:

[0095]

[0096] In knowledge distillation training, the true label is the hard label, the probability value output by the teacher model is the soft label, the output of the student model and the hard label are used to calculate the cross entropy loss, and the soft label is used to calculate the mean square error loss. The weighted sum of the cross entropy loss and the mean square error loss is used as the final loss function. The loss function formula is:

[0097]

[0098]

[0099] L=αL ce_loss +(1-α)L mse_loss

[0100] Where: N represents the number of samples, M represents the number of categories, the base of log is e, y ic Indicates the true category of sample i, c takes 1 or 0, p ic It represents the probability that the predicted sample i belongs to category c. α needs to be set to an initial value and is obtained through multiple rounds of learning.

[0101] After knowledge distillation is completed, the model is compressed again by reducing the number of network layers through pruning, converting the 32-bit floating-point parameter values ​​into int8 through quantization, and fine-tuning the technology again to reduce the size of the model without losing accuracy as much as possible. This improves the efficiency of the translation model and enables high concurrency and high performance support for the translation model.< / end> < / start>

Claims

1. A medical machine translation method based on reinforcement learning, characterized by: The following steps are involved: Step 1: Construct a parallel corpus: The parallel corpus includes a general corpus sub-corpus and a medical corpus sub-corpus. The general corpus sub-corpus and the medical corpus sub-corpus have the same structure and both include one or more parallel corpora. The parallel corpora consist of the first language, the first language text, the second language and the second language text. The length of the parallel corpus meets the preset length range. Step 2: Build a machine translation model: The machine translation model includes an encoder and a decoder. The decoder has the same number of layers as the encoder. The output of the encoder is connected to the corresponding input of the decoder. Step 3: Pre-training the machine translation model: extracting corpus from the general corpus sub-database and the medical corpus sub-database according to a preset ratio to construct a basic parameter training set, inputting the basic parameter training set into the machine translation model to train the basic parameters of the machine translation model; Step 4: Calculate the translation quality score: Extract corpus from the general corpus sub-corpus and the medical corpus sub-corpus according to a preset ratio to construct a test set. Input the first language, first language text, and second language of the test set into the machine translation model trained in step 3 to obtain the translated text. Compare the translated text with the second language text and calculate the BLEU (Bilingual Evaluation Understudy) score. Step 5: Construct a reinforcement learning training set: Filter the parallel corpora in the test set whose BLEU scores are lower than the preset threshold score and use them as the basic corpora; process the basic corpora one by one to construct the reinforcement learning training set; The specific steps include: Step 5-1: Construct pseudo-parallel corpus: Use the following three methods to construct pseudo-parallel corpus corresponding to the basic prediction: Method 1: Divide the first language text in the basic corpus into more than one word, randomly extract n words from them, where n>1, and replace them with corresponding synonyms; Method 2: Translate the second language text of the basic corpus into a language different from the first and second languages, and then translate it back to the second language; Method 3: Train the mt5 model to obtain a sentence rewriting model; input the basic corpus into the trained sentence rewriting model to obtain a pseudo-parallel corpus; Step 5-2: Use cosine similarity to score each pseudo-parallel corpus, remove those with scores below a preset threshold and those that are completely identical to the original text, and add the remaining pseudo-parallel corpora to the reinforcement learning training set; Step 6: Fine-tune the machine translation model: Use the reinforcement learning training set to fine-tune the machine translation model for one or more rounds to obtain the optimized parameters of the machine translation model.

2. The medical machine translation method based on reinforcement learning according to claim 1, characterized in that: The term intervention steps is also included, including the following specific steps: Step A-1: ​​Create a medical terminology glossary with the data format being [English, Chinese, translation direction], where the translation direction is English to Chinese or Chinese to English; Step A-2: searching for medical terms in the text to be translated, and replacing the medical terms in the text to be translated with placeholders; Search the medical terminology glossary to obtain the corresponding translation terms; Step A-3: using the machine translation model to translate the text to be translated processed in step A-2; Step A-4: Replace the placeholders in the translation text obtained in step A-3 with the translated words.

3. The medical machine translation method based on reinforcement learning according to claim 1, characterized in that: It also includes a model compression step, including the following specific steps: Step A: Generate the predicted probability of each target word through the softmax function: Where j = 1, 2, ..., K, z is a vector; T is the temperature coefficient, greater than 1; Step B: Knowledge distillation training: The trained model is used as the teacher model, and the model with reduced model layers is used as the student model. The true label is the hard label, and the predicted probability value output by the teacher model is the soft label. The cross entropy loss is calculated for the output of the student model and the hard label, and the mean squared error loss is calculated for the soft label. The weighted sum of the cross entropy loss and the mean squared error loss is used as the final loss function. L=αL ce_loss +(1-α)L mse_loss Where: N represents the number of samples, M represents the number of categories, log is the logarithm operation with the base e, indicating whether sample i belongs to category c, which takes 1 or 0, indicating the probability of predicting that sample i belongs to category c, which is the weight.

4. The medical machine translation method based on reinforcement learning according to claim 3, characterized in that: The method also includes a pruning step: reducing the number of network layers to a preset number of layers, reducing the accuracy of the parameters to a preset number of bits by quantization, and optimizing the student model parameters using step 5 again.

5. The medical machine translation method based on reinforcement learning according to claim 1, characterized in that: The machine translation model includes an input layer, a convolutional neural network, an encoder, and a decoder. The encoder consists of an input layer and the first to 12 encoder layers with the same structure, which are cascaded in sequence. The decoder consists of an input layer and the first to 12 decoder layers with the same structure, which are cascaded in sequence. The convolutional neural network consists of a convolutional layer, a pooling layer, and an output layer, which are cascaded in sequence. The input layer structure of the encoder and decoder is the same, and both consist of an embedding sublayer and a position sublayer. Each encoder layer consists of a multi-head self-attention sublayer, a first residual sublayer, a feedforward network sublayer, and a second residual sublayer cascaded in sequence. After word segmentation of the first language text, the first language word sequence [x1, x2, ..., xm] is obtained. The first language word sequence [x1, x2, ..., xm] is input into the embedding sublayer and the position sublayer respectively. The outputs of the embedding sublayer and the position sublayer are added and then input into the multi-head self-attention sublayer and the first residual sublayer of the first encoder layer respectively. The output end of the first residual sublayer of the first encoder layer is connected to the corresponding input end of the second residual sublayer of the first encoder layer, and the output end of the second residual sublayer is used as the output of the first encoder layer. Each decoder layer consists of a multi-head self-attention sublayer, a first residual sublayer, an encoder-decoder attention sublayer, a second residual sublayer, a feedforward network sublayer, and a third residual sublayer cascaded in sequence. The output end of the first residual sublayer of the first decoder layer is connected to the corresponding input end of the second residual sublayer of the first decoder layer, the corresponding input end of the second residual sublayer of the first decoder layer is connected to the corresponding input end of the third residual sublayer of the first decoder layer, and the output end of the third residual sublayer of the first decoder layer is used as the output of the first decoder layer; The output word encoding information matrix C of the 12th layer encoder is added to the output semantic information matrix T of the convolutional neural network to obtain the vector matrix S: S=C+αT Where, α is the sentence semantic information weight; Input the vector matrix S to the encoder-decoder attention sublayer of each decoder layer; The starting character START is input into the embedding sublayer and position sublayer of the decoder, and is processed by the 1st to 12th decoder layers to obtain the predicted value y1. The predicted value y1 is fed back into the embedding sublayer and position sublayer to obtain the predicted word y2. The predicted value yi, i>1, is fed back into the embedding sublayer and position sublayer to obtain the predicted value yi+1, until the predicted value yi+1 is the end symbol END.

Images