Translation method based on omnidirectional attention mechanism

Abstract

The invention discloses a translation method based on an omnidirectional attention mechanism, which relates to the technical field of natural language processing, and comprises the following steps: collecting and processing parallel corpus data, generating a distillation data set through knowledge distillation, training an autoregression translation model by utilizing the distillation data set, and solving a multi-mode problem. The autoregression model is converted into a non-autoregression model, training is conducted till convergence is achieved, the omni-directional attention mechanism and the course learning strategy are introduced, the multi-mode problem occurring in the non-autoregression translation model is effectively solved, the translation quality and training efficiency are remarkably improved, and therefore the more accurate translation output effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a translation method based on an omnidirectional attention mechanism. Background Art

[0002] With the development of information technology, natural language processing, as an important branch of artificial intelligence, has been increasingly studied and applied. Traditional statistical machine translation models are unable to cope with the demand due to their reliance on complex feature engineering, while neural machine translation based on deep learning has achieved a leap in translation performance through end-to-end learning.

[0003] In the prior art, knowledge distillation is a commonly used method that improves the translation quality of non-autoregressive models by transferring the knowledge of autoregressive models to non-autoregressive models. In the process of knowledge distillation, the target sequence generated by the autoregressive model is usually regarded as a "soft label", while the target sequence generated by the non-autoregressive model is regarded as a "hard label". This practice results in the non-autoregressive model being unable to generate high-quality translation results in actual situations. Therefore, how to use the omnidirectional attention reasoning module to adjust the teacher model through the curriculum learning strategy to solve the multi-modal problem has become the key to improving the performance of non-autoregressive translation models. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a translation method based on an omnidirectional attention mechanism to solve the multi-modal problem of the autoregressive translation model and to improve the translation accuracy and speed of the non-autoregressive model.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, an embodiment of the present invention provides a translation method based on an omnidirectional attention mechanism, which includes collecting and processing parallel corpus data; Use the processed parallel corpus data and the autoregressive translation model with the same parameter scale as the non-autoregressive translation model to perform knowledge distillation and generate a distilled dataset; Use the distilled dataset to train an autoregressive translation model to identify the multimodal issues that exist; Propose improvement ideas and solutions based on existing multi-mode problems; After proposing improvement ideas and solutions, the target language generated by the autoregressive translation model is converted into the target language generated by the non-autoregressive translation model; After completing the conversion from the autoregressive translation model to the non-autoregressive translation model, the non-autoregressive translation model is trained using the distilled dataset until convergence; After the non-autoregressive translation model converges, the non-autoregressive translation model is evaluated and optimized.

[0007] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, wherein: collecting parallel corpus data and processing it includes the following steps: Collect parallel corpora in the source and target languages ​​that have been correctly translated and strictly aligned; Clean the collected parallel corpus, unify the case of sentences, standardize punctuation, remove unnecessary marks and special characters, and filter out sentences that are too long, too short, and have serious errors; A vocabulary is constructed based on the cleaned parallel corpus, and the sentences in the vocabulary are segmented.

[0008] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, wherein: using the processed parallel corpus data and an autoregressive translation model with a parameter scale equivalent to that of the non-autoregressive translation model to perform knowledge distillation to generate a distilled data set, the method comprises the following steps: Select an autoregressive model with excellent performance as the teacher model; The teacher model is used to perform forward propagation and knowledge distillation on the processed parallel corpus data to generate the character sequence of the target sentence and obtain a pure distilled data set.

[0009] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, wherein: using the distillation data set to train the autoregressive translation model to find out the existing multi-modal problems, the following steps are included: After generating the distilled dataset, the student model is trained based on the generated distilled dataset while learning the hard labels of the original training data to find multimodal problems; According to the multi-modal problem, a comprehensive loss function is defined. One part is the prediction error of the student model for the original label, and the other part is the gap between the output of the student model and the soft label of the teacher model. The expression of the comprehensive loss function of the student model is: ; in, represents the comprehensive loss function, represents the loss term on the original training dataset, and Respectively represent the balance of the impact of different loss terms and The weight coefficient of represents the loss term on the distilled dataset, Represents the loss term during the transition from an autoregressive model to a non-autoregressive model.

[0010] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, according to the existing multi-modal problems, an improvement idea and solution are proposed, including the following steps: According to the existing multi-modal problem, an omnidirectional attention reasoning module is introduced to enable information at all positions to be observed when generating character sequences; The teacher model is gradually adjusted through the curriculum learning strategy, so that the teacher model can smoothly transition to the student model.

[0011] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, after proposing the improvement ideas and solutions, the target language generated by the autoregressive translation model is converted into the target language generated by the non-autoregressive translation model, including the following steps: When starting the conversion, the target language information generated by the autoregressive translation model must be consistent with the target language information of the previously generated character sequence; At each stage of teacher model training, the substitution rate is calculated to evaluate the performance of the non-autoregressive translation model; The expression for calculating the replacement rate of teacher model training is: ; in, represents the replacement rate, represents the total number of training rounds, Indicates the number of sentences involved in the calculation of entropy and hash function values, represents the sentence length of the dataset, Indicates Sentence length The information entropy calculation is Indicates the hash function value corresponding to the sentence length, It represents the evaluation value of sentence length by the normal distribution density function with the mean and variance of sentence length as parameters. represents the total number of sentences in the distillation dataset, Represents the first The length of the sentence, Indicates Distilled data sentence length The function value of represents the logarithmic function value of the sentence length, Represents the slight change in the length of each sentence in the distilled dataset.

[0012] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, after completing the conversion from the autoregressive translation model to the non-autoregressive translation model, the non-autoregressive translation model is trained using the distilled data set until convergence, including the following steps: Before using the distilled dataset to train the non-autoregressive translation model, a neural network architecture for the non-autoregressive task is selected, and the non-autoregressive model is randomly initialized and pre-trained using the neural network architecture for the non-autoregressive task; During training, the non-autoregressive model uses a cross-entropy loss to minimize the difference between the predicted target sequence and the target sequence provided by the teacher model in the distilled dataset; When generating the final output, a decoding strategy using post-processing technology is used to remove the difference between the minimized predicted target sequence in the non-autoregressive model and the target sequence provided by the teacher model in the distillation dataset to improve the translation quality. At the same time, an auxiliary loss of consistency loss needs to be added to improve the performance of the non-autoregressive model to ensure that the non-autoregressive model can produce the same output for different noise inputs, thereby completing the convergence of the autoregressive translation model.

[0013] As a preferred solution of the translation method based on the omnidirectional attention mechanism described in the present invention, wherein: evaluating and optimizing the non-autoregressive translation model comprises the following steps: Evaluate and test the performance of non-autoregressive translation models on a distillation dataset; Tune the learning rate and batch size hyperparameters of the non-autoregressive translation model based on the evaluation performance on the distillation dataset, and use model fusion with different initializations; After improving the performance of the non-autoregressive translation model, we exported the model into an easy-to-deploy format and optimized the model by using quantization and pruning techniques to further increase the inference speed and reduce memory usage. After the non-autoregressive translation model is optimized, a real-time monitoring system is established to monitor the performance indicators of the response time and throughput of the non-autoregressive translation model in the production environment. At the same time, an anomaly detection mechanism is set up to promptly discover and handle specific problems encountered by the model in actual application; Based on feedback from actual applications, the non-autoregressive translation model is retrained regularly and irregularly to adapt to the changing data distribution.

[0014] In a second aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the translation method based on the omnidirectional attention mechanism as described in the first aspect of the present invention is implemented.

[0015] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the translation method based on the omnidirectional attention mechanism as described in the first aspect of the present invention is implemented.

[0016] The beneficial effects of the present invention are as follows: by collecting and processing parallel corpus data, the preparation of high-quality training data is achieved, the accuracy and fluency of translation are improved, the autoregressive model is used to perform knowledge distillation on the processed corpus, ensuring that the non-autoregressive model can obtain high-quality knowledge transfer, thereby improving the learning efficiency and generalization ability of the translation model, using the distilled data set to train the autoregressive model and identify multimodal problems, achieving problem location and improvement in the model training process, ensuring that the model can maintain high translation quality in complex scenarios, introducing an omnidirectional attention mechanism and gradually adjusting the teacher model to the student model, solving multimodal problems, enhancing the model's ability to understand the context during translation, and achieving a more natural and accurate translation output effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 This is a flowchart of the translation method based on the omnidirectional attention mechanism in Example 1.

[0019] Figure 2 This is the decision diagram of the translation method based on the omnidirectional attention mechanism in Example 1. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a translation method based on an omnidirectional attention mechanism, comprising the following steps: S1. Collect and process parallel corpus data, including the following steps: Collect parallel corpora in the source and target languages ​​that have been correctly translated and strictly aligned; Parallel corpus refers to a data set containing corresponding texts in two or more languages. The corresponding texts in different languages ​​match each other. An article and a sentence have accurate translations in different language versions. Since parallel corpus is carefully translated and proofread by professional translators, the translation quality is relatively high. Parallel corpus is usually used as a benchmark for training high-quality machine translation systems. Clean the collected parallel corpus, unify the case of sentences, standardize punctuation, remove unnecessary marks and special characters, and filter out sentences that are too long, too short, and have serious errors; A vocabulary is constructed based on the cleaned parallel corpus, and the sentences in the vocabulary are segmented.

[0024] A vocabulary is an ordered set of words used to map words in natural language to numerical representations so that machine learning models can process words in natural language and convert words in text into corresponding index numbers. For example, the source language sentence "Hello, how are you?" can be encoded as [2, 6, 3, 4, 5, 7].

[0025] S2. Use the processed parallel corpus data and the autoregressive translation model with the same parameter scale as the non-autoregressive translation model to perform knowledge distillation to generate a distilled data set, including the following steps: Select an autoregressive model with excellent performance as the teacher model; Use the teacher model to perform forward propagation and knowledge distillation on the processed parallel corpus data to generate the character sequence of the target sentence and obtain a pure distilled data set; Forward propagation is a basic operation process in neural networks, which describes how input data passes through each layer of the network and finally produces output. In a neural network, the output of each layer is a function of the output of the previous layer. This process continues until the last output layer produces the final prediction result. It is also the basis for using the trained model for prediction. Knowledge distillation technology is a model compression method that aims to transfer the knowledge of the teacher model to the student model. This technology is particularly popular in the field of deep learning because it can significantly reduce the size and computational cost of the model while maintaining model performance. In machine translation tasks, a complex autoregressive model is used as the teacher model, and a non-autoregressive model is trained as the student model through knowledge distillation technology to improve translation speed while maintaining translation quality. In image recognition tasks, a large convolutional neural network is used as the teacher model, and a MobileNet network is trained through knowledge distillation technology to achieve higher computational efficiency. S3. Use the distilled dataset to train the autoregressive translation model to find out the multimodal problems, including the following steps: After generating the distilled dataset, the student model is trained based on the generated distilled dataset while learning the hard labels of the original training data to find multimodal problems; The multimodal problem refers to the situation in which, in machine translation tasks, given a source language sentence, there are multiple different target language translations. For example, translating "Thank you" into "Danke Dank" is unreasonable in the actual language. This problem is particularly prominent in non-autoregressive translation models because they lack contextual dependencies when generating target sequences, resulting in poor diversity and coherence in the generated results. For example, "I like to eat apples" may be translated into "Ich mag Äpfel essen zu" instead of the more reasonable "Ich mag Äpfelessen"; According to the multi-modal problem, a comprehensive loss function is defined. One part is the prediction error of the student model for the original label, and the other part is the gap between the output of the student model and the soft label of the teacher model. The expression of the comprehensive loss function of the student model is: ; in, represents the comprehensive loss function, represents the loss term on the original training dataset, and Respectively represent the balance of the impact of different loss terms and The weight coefficient of represents the loss term on the distilled dataset, Represents the loss term during the transition from an autoregressive model to a non-autoregressive model.

[0026] S4. Based on the existing multi-mode problems, propose improvement ideas and solutions, including the following steps: According to the existing multi-modal problem, an omnidirectional attention reasoning module is introduced to enable information at all positions to be observed when generating character sequences; The omnidirectional attention reasoning module is an improved attention mechanism. In order to solve the multi-modal problem and insufficient context dependency faced by non-autoregressive models when generating sequences, the omnidirectional attention reasoning module enables the non-autoregressive model to access information at all positions when generating each target word, thereby better utilizing context information and improving the coherence and accuracy of the generated sequence. Gradually adjust the teacher model through the course learning strategy to make the teacher model smoothly transition to the student model; Curriculum learning is a training strategy that gradually adjusts the difficulty or conditions during the training process so that the model can better learn and adapt to the task. In the process of knowledge distillation, the curriculum learning strategy is used to gradually adjust the way the teacher model guides the student model, thereby helping the student model to better learn the knowledge of the teacher model. At each adjustment stage, the performance of the student model is regularly evaluated to ensure that the student model can gradually learn to independently generate high-quality outputs. If the student model performs poorly at a certain stage, the decay rate of the replacement rate and the final replacement rate value can be appropriately adjusted.

[0027] S5. After proposing improvement ideas and solutions, the target language generated by the autoregressive translation model is converted into the target language generated by the non-autoregressive translation model, including the following steps: When starting the conversion, the target language information generated by the autoregressive translation model must be consistent with the target language information of the previously generated character sequence; At each stage of teacher model training, the substitution rate is calculated to evaluate the performance of the non-autoregressive translation model; The substitution rate is a concept used in the course learning strategy. It refers to the ratio of the output of the teacher model to the output generated by the student model itself when training the student model. The specific value is between 0 and 1. It controls the degree to which the student model depends on the teacher model during the training process. When the substitution rate is 1, it means that the student model completely depends on the output of the teacher model; when the substitution rate is 0, it means that the student model generates output completely independently. The expression for calculating the replacement rate of teacher model training is: ; in, represents the replacement rate, represents the total number of training rounds, Indicates the number of sentences involved in the calculation of entropy and hash function values, represents the sentence length of the dataset, Indicates Sentence length The information entropy calculation is Indicates the hash function value corresponding to the sentence length, It represents the evaluation value of sentence length by the normal distribution density function with the mean and variance of sentence length as parameters. represents the total number of sentences in the distillation dataset, Represents the first The length of the sentence, Indicates Distilled data sentence length The function value of represents the logarithmic function value of the sentence length, Represents the slight change in the length of each sentence in the distilled dataset.

[0028] S6. After completing the conversion from the autoregressive translation model to the non-autoregressive translation model, the non-autoregressive translation model is trained using the distilled dataset until convergence, including the following steps, Before using the distilled dataset to train the non-autoregressive translation model, a neural network architecture for the non-autoregressive task is selected, and the non-autoregressive model is randomly initialized and pre-trained using the neural network architecture for the non-autoregressive task; Random initialization and pre-trained initialization are two common initialization methods in the deep learning model training process. They are suitable for different scenarios and have their own advantages and disadvantages. Random initialization means that the weights and biases of the model are randomly assigned before model training begins. The advantage of random initialization is that it is very flexible and applicable to any model architecture, and does not require additional data and pre-trained models. The disadvantage is that it causes the model to converge slowly in the early stages of training in deep networks. Pre-training initialization means that before model training begins, the model weights are initialized to the weights of another trained model. The advantages of pre-training initialization can help the model converge faster, have stronger generalization ability, and help reduce overfitting; During training, the non-autoregressive model uses a cross-entropy loss to minimize the difference between the predicted target sequence and the target sequence provided by the teacher model in the distilled dataset; Cross entropy loss refers to measuring the difference between two probability distributions in multi-classification problems, so that the model can better fit the training data. When the probability distribution predicted by the model is consistent with the true label distribution, the cross entropy loss is minimal. During the training process, by minimizing the cross entropy loss, the model can learn a predicted probability distribution that is closer to the true label. When the model predicts incorrectly, the cross entropy loss will increase significantly, prompting the model to adjust parameters to reduce this error.

[0029] When generating the final output, a decoding strategy using post-processing technology is used to remove the difference between the minimized predicted target sequence in the non-autoregressive model and the target sequence provided by the teacher model in the distillation dataset to improve the translation quality. At the same time, an auxiliary loss of consistency loss needs to be added to improve the performance of the non-autoregressive model to ensure that the non-autoregressive model can produce the same output for different noise inputs, thereby completing the convergence of the autoregressive translation model.

[0030] The decoding strategy of post-processing technology refers to the process of further optimizing the generated results through a series of technologies and methods after generating the output of the machine translation model. By correcting errors and unreasonable places in the generated results, it ensures that the generated sentences are more coherent and natural in terms of grammar and semantics.

[0031] S7. Evaluate and optimize the non-autoregressive translation model, including the following steps: Evaluate and test the performance of non-autoregressive translation models on a distillation dataset; Tune the learning rate and batch size hyperparameters of the non-autoregressive translation model based on the evaluation performance on the distillation dataset, and use model fusion with different initializations; Hyperparameters are pre-set parameters in machine learning and deep learning models, which are manually set by developers and researchers. Choosing appropriate hyperparameters can significantly improve the performance of the model, while poor hyperparameter settings can lead to poor model performance. The learning rate hyperparameter determines the magnitude of the model parameter update in each iteration. A higher learning rate will make the training process faster, but will lead to unstable model convergence. A lower learning rate will make the model converge more stably, but the training process will be slower. Common learning rate adjustment strategies include fixed learning rate, exponential decay, and adaptive learning rate in Adam optimizer. The batch size hyperparameter determines the number of samples used when updating model parameters. A larger batch size will make each gradient update closer to the true gradient, but requires more memory. A smaller batch size will make the training process more flexible, but will cause fluctuations in gradient estimates. Common batch sizes are 32, 64, and 128. After improving the performance of the non-autoregressive translation model, we exported the model into an easy-to-deploy format and optimized the model by using quantization and pruning techniques to further increase the inference speed and reduce memory usage. Quantization refers to the process of converting the weights and activation values ​​in the model from 32-bit floating point numbers to 8-bit integers. Through quantization, the storage space requirements and computational overhead of the model can be significantly reduced; Pruning is the process of reducing the number of model parameters and computational complexity by removing unimportant weights or channels in the model. Pruning can reduce the memory bandwidth required by the model at runtime, thereby improving computational efficiency. After the non-autoregressive translation model is optimized, a real-time monitoring system is established to monitor the performance indicators of the response time and throughput of the non-autoregressive translation model in the production environment. At the same time, an anomaly detection mechanism is set up to promptly discover and handle specific problems encountered by the model in actual application; Based on feedback from actual applications, the non-autoregressive translation model is retrained regularly and irregularly to adapt to the changing data distribution.

[0032] This embodiment also provides a computer device, which is suitable for the case of a translation method based on an omnidirectional attention mechanism, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the translation method based on the omnidirectional attention mechanism proposed in the above embodiment. The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0033] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the translation method based on the omnidirectional attention mechanism proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0034] In summary, the present invention collects and processes parallel corpus data to prepare high-quality training data, improves the accuracy and fluency of translation, uses an autoregressive model to perform knowledge distillation on the processed corpus, ensures that the non-autoregressive model can obtain high-quality knowledge transfer, and thus improves the learning efficiency and generalization ability of the translation model, uses a distilled data set to train the autoregressive model and identify multimodal problems, realizes problem location and improvement in the model training process, ensures that the model can maintain high translation quality in complex scenarios, introduces an omnidirectional attention mechanism and gradually adjusts the teacher model to the student model, solves the multimodal problem, enhances the model's ability to understand the context during translation, and achieves a more natural and accurate translation output effect.

[0035] Example 2, referring to Table 1, is the second example of the present invention. In order to further verify the technical solution of the present invention, experimental simulation data of a translation method based on an omnidirectional attention mechanism is provided.

[0036] Experimental preparation and implementation process In order to verify the effectiveness of the translation method based on the omnidirectional attention mechanism, the following experimental steps were carried out: Data Collection and Processing: Collect an English-Chinese parallel corpus containing 100,000 correctly translated and strictly aligned sentences.

[0037] The collected parallel corpus was cleaned, including unifying capitalization, normalizing punctuation, removing unnecessary tags and special characters, and filtering out sentences that were too long (more than 100 words) and too short (less than 5 words) as well as sentences with serious errors.

[0038] Build a vocabulary and perform word segmentation on the sentences in the vocabulary.

[0039] Knowledge distillation preparation: Select an autoregressive model that performs well on the English-Chinese translation task as the teacher model.

[0040] The teacher model is used to perform forward propagation and knowledge distillation on the processed parallel corpus data to generate the character sequence of the target sentence and obtain a pure distilled data set.

[0041] Multi-modal problem identification and improvement: The distilled dataset is used to train a student model with comparable parameter size to a non-autoregressive translation model.

[0042] During the training process, a comprehensive loss function is defined, which includes the loss term on the original training dataset, the loss term on the distilled dataset, and the loss term during the conversion from the autoregressive model to the non-autoregressive model.

[0043] By introducing the omnidirectional attention reasoning module, information at all positions can be observed when generating character sequences.

[0044] The teacher model is gradually adjusted through the curriculum learning strategy, so that the teacher model can smoothly transition to the student model.

[0045] Conversion and training: At each teacher model training stage, the substitution rate is calculated to evaluate the performance of the non-autoregressive translation model.

[0046] After completing the conversion from the autoregressive translation model to the non-autoregressive translation model, the non-autoregressive translation model is trained using the distilled dataset until convergence.

[0047] Select a neural network architecture for a non-autoregressive task and use the neural network architecture for the non-autoregressive task to perform random initialization and pre-training initialization on the non-autoregressive model.

[0048] During training, the non-autoregressive model uses a cross-entropy loss to minimize the difference between the predicted target sequence and the target sequence provided by the teacher model in the distilled dataset.

[0049] The decoding strategy using post-processing technology removes the difference between the minimized predicted target sequence in the non-autoregressive model and the target sequence provided by the teacher model in the distillation dataset to improve the translation quality, and adds an auxiliary loss of consistency loss to improve the performance of the non-autoregressive model.

[0050] Evaluation and Optimization: Evaluate and test the performance of non-autoregressive translation models on the distillation dataset.

[0051] The learning rate and batch size hyperparameters of the non-autoregressive translation model are adjusted based on the evaluation performance, and the models with different initializations are fused.

[0052] After improving the performance of the non-autoregressive translation model, we exported the model into an easy-to-deploy format and used quantization and pruning techniques to further improve inference speed and reduce memory usage.

[0053] Establish a real-time monitoring system to monitor the performance indicators of response time and throughput of the non-autoregressive translation model in the production environment, and set up an anomaly detection mechanism to promptly discover and handle specific problems encountered by the model in actual application.

[0054] Based on feedback from real applications, the non-autoregressive translation model is retrained periodically and irregularly to adapt to the changing data distribution.

[0055] The details are shown in Table 1 below: Table 1 Performance comparison of translation methods based on omnidirectional attention mechanism

[0056] It can be seen from the above table data that the method of the present invention has significant advantages over the traditional neural machine translation method: BLEU Score (bilingual evaluation index): The BLEU score of the proposed method before optimization was 28.5, which was significantly higher than the 25.3 of the traditional NMT. ​​The BLEU score after optimization reached 32.1, indicating that the method has significantly improved the translation quality.

[0057] Perplexity: The perplexity of the proposed method before optimization was 9.8, which is lower than the 11.4 of traditional NMT. ​​The perplexity after optimization was further reduced to 8.5, indicating that the model is more accurate in predicting the next word and the overall translation quality is higher.

[0058] Training Time: The training time of the proposed method before optimization was 9 hours, which was shorter than the 10 hours of traditional NMT. ​​After optimization, the training time was further shortened to 7 hours, indicating that the training efficiency of the model has been significantly improved.

[0059] Memory Usage: The memory usage of the proposed method before optimization is 1400MB, which is lower than the 1500MB of traditional NMT. ​​After optimization, the memory usage is further reduced to 1000MB, indicating that the model is more economical in resource usage and more suitable for deployment in resource-constrained environments.

[0060] In summary, the method of the present invention effectively solves the multi-modal problem existing in the traditional non-autoregressive translation model by introducing the omnidirectional attention mechanism, knowledge distillation technology and curriculum learning strategy, improves the translation quality and training efficiency, and makes the non-autoregressive translation model more efficient and reliable in actual deployment.

[0061] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A translation method based on omnidirectional attention mechanism, Features: include, Collect and process parallel corpus data; Use the processed parallel corpus data and the autoregressive translation model with the same parameter scale as the non-autoregressive translation model to perform knowledge distillation and generate a distilled dataset; Use the distilled dataset to train an autoregressive translation model to identify the multimodal issues that exist; Propose improvement ideas and solutions based on existing multi-mode problems; After proposing improvement ideas and solutions, the target language generated by the autoregressive translation model is converted into the target language generated by the non-autoregressive translation model; After completing the conversion from the autoregressive translation model to the non-autoregressive translation model, the non-autoregressive translation model is trained using the distilled dataset until convergence; Evaluate and optimize non-autoregressive translation models.

2. The translation method based on the omnidirectional attention mechanism as claimed in claim 1, It is characterized in that Collect and process parallel corpus data, The following steps are included: Collect parallel corpora in the source and target languages ​​that have been correctly translated and strictly aligned; Clean the collected parallel corpus, unify the case of sentences, standardize punctuation, remove unnecessary marks and special characters, and filter out sentences that are too long, too short, and have serious errors; A vocabulary is constructed based on the cleaned parallel corpus, and the sentences in the vocabulary are segmented.

3. The translation method based on the omnidirectional attention mechanism as claimed in claim 2, It is characterized in that Using the processed parallel corpus data and an autoregressive translation model with the same parameter scale as the non-autoregressive translation model to perform knowledge distillation, a distillation dataset is generated, including the following steps: Select an autoregressive model with excellent performance as the teacher model; The teacher model is used to perform forward propagation and knowledge distillation on the processed parallel corpus data to generate the character sequence of the target sentence and obtain a pure distilled data set.

4. The translation method based on the omnidirectional attention mechanism as claimed in claim 3, It is characterized in that Using the distilled dataset to train an autoregressive translation model to find out the multimodal issues, the following steps are included: After generating the distilled dataset, the student model is trained based on the generated distilled dataset while learning the hard labels of the original training data to find multimodal problems; According to the multi-modal problem, a comprehensive loss function is defined. One part is the prediction error of the student model for the original label, and the other part is the gap between the output of the student model and the soft label of the teacher model. The expression of the comprehensive loss function of the student model is: ； in, represents the comprehensive loss function, represents the loss term on the original training dataset, and Respectively represent the balance of the impact of different loss terms and The weight coefficient of represents the loss term on the distilled dataset, Represents the loss term during the transition from an autoregressive model to a non-autoregressive model.

5. The translation method based on the omnidirectional attention mechanism as claimed in claim 4, It is characterized in that According to the existing multi-mode problems, improvement ideas and solutions are proposed, including the following steps: According to the existing multi-modal problem, an omnidirectional attention reasoning module is introduced to enable information at all positions to be observed when generating character sequences; The teacher model is gradually adjusted through the curriculum learning strategy, so that the teacher model can smoothly transition to the student model.

6. The translation method based on the omnidirectional attention mechanism as claimed in claim 5, It is characterized in that After proposing improvement ideas and solutions, the target language generated by the autoregressive translation model is converted into the target language generated by the non-autoregressive translation model, including the following steps: When starting the conversion, the target language information generated by the autoregressive translation model must be consistent with the target language information of the previously generated character sequence; At each stage of teacher model training, the substitution rate is calculated to evaluate the performance of the non-autoregressive translation model; The expression for calculating the replacement rate of teacher model training is: ； in, represents the replacement rate, represents the total number of training rounds, Indicates the number of sentences involved in the calculation of entropy and hash function values, represents the sentence length of the dataset, Indicates Sentence length The information entropy calculation is Indicates the hash function value corresponding to the sentence length, It represents the evaluation value of sentence length by the normal distribution density function with the mean and variance of sentence length as parameters. represents the total number of sentences in the distillation dataset, Represents the first The length of the sentence, Indicates Distilled data sentence length The function value of represents the logarithmic function value of the sentence length, Represents the slight change in the length of each sentence in the distilled dataset.

7. The translation method based on the omnidirectional attention mechanism as claimed in claim 6, It is characterized in that After completing the conversion from the autoregressive translation model to the non-autoregressive translation model, the non-autoregressive translation model is trained using the distilled dataset until convergence, including the following steps: Before using the distilled dataset to train the non-autoregressive translation model, a neural network architecture for the non-autoregressive task is selected, and the non-autoregressive model is randomly initialized and pre-trained using the neural network architecture for the non-autoregressive task; During training, the non-autoregressive model uses a cross-entropy loss to minimize the difference between the predicted target sequence and the target sequence provided by the teacher model in the distilled dataset; When generating the final output, a decoding strategy using post-processing technology is used to remove the difference between the minimized predicted target sequence in the non-autoregressive model and the target sequence provided by the teacher model in the distillation dataset to improve the translation quality. At the same time, an auxiliary loss of consistency loss needs to be added to improve the performance of the non-autoregressive model to ensure that the non-autoregressive model can produce the same output for different noise inputs, thereby completing the convergence of the autoregressive translation model.

8. The translation method based on the omnidirectional attention mechanism as claimed in claim 7, It is characterized in that Evaluate and optimize non-autoregressive translation models, The following steps are included: Evaluate and test the performance of non-autoregressive translation models on a distillation dataset; Tune the learning rate and batch size hyperparameters of the non-autoregressive translation model based on the evaluation performance on the distillation dataset, and use model fusion with different initializations; After improving the performance of the non-autoregressive translation model, we exported the model into an easy-to-deploy format and optimized the model by using quantization and pruning techniques to further increase the inference speed and reduce memory usage. After the non-autoregressive translation model is optimized, a real-time monitoring system is established to monitor the performance indicators of the response time and throughput of the non-autoregressive translation model in the production environment. At the same time, an anomaly detection mechanism is set up to promptly discover and handle specific problems encountered by the model in actual application; Based on feedback from actual applications, the non-autoregressive translation model is retrained regularly and irregularly to adapt to the changing data distribution.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. Features: When the processor executes the computer program, the steps of the translation method based on the omnidirectional attention mechanism described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, Features: When the computer program is executed by a processor, the steps of the translation method based on the omnidirectional attention mechanism described in any one of claims 1 to 8 are implemented.

Images