Neural machine translation selective knowledge distillation method based on dependency constraint self-attention

Abstract

The invention relates to a neural machine translation selective knowledge distillation method based on dependency constraint self-attention, and belongs to the technical field of machine translation. An existing knowledge distillation method has the problems that only vocabulary-level probability distribution is transmitted, syntactic structure constraints are ignored, and the capacity of a student model is reduced, so that the complex syntactic modeling capacity is insufficient. Therefore, according to the method, a syntactic matrix converted through a source language dependency syntactic tree is provided, linear combination is adopted to dynamically correct self-attention weight distribution of an encoder, and explicit syntactic constraints are synchronously injected into a teacher-student model. Through a selective distillation strategy of syntax perception, deep knowledge effective for training in a teacher model is screened, a cross-language syntax corresponding relation is obtained in combination with structure alignment distillation, and model compression and translation performance enhancement is realized through attention optimization and a selective knowledge transmission mechanism guided by a dependency syntax.

Description

TECHNICAL FIELD

[0001] The present application relates to a neural machine translation selective knowledge distillation method based on dependency constraint self-attention, and belongs to the field of machine translation. BACKGROUND

[0002] In recent years, neural machine translation has made significant progress in the field of translation, especially the introduction of the Transformer architecture makes the model perform well in the translation task. However, such models have extremely high requirements for storage and computing resources, and face challenges in deployment in practical applications. To solve this problem, many methods use knowledge distillation methods for model compression, which pass the knowledge of a complex teacher model to a simplified student model, and realize model lightweight on the premise of not losing performance as much as possible. Existing knowledge distillation techniques are usually divided into two categories: word-level knowledge distillation and sequence-level knowledge distillation. Word-level knowledge distillation passes the output probability distribution of the teacher model to the student model as soft labels, and uses KL divergence to measure the difference between the output probability distributions as the loss function. Sequence-level knowledge distillation focuses on global sequence prediction and learns the complete sequence output distribution of the teacher model. For example, when processing the sentence "The boy whom I met yesterday is crying" containing complex dependency structure, the teacher model can capture the dependency relationship between "whom" and "met" through self-attention mechanism, and generate the correct translation "Yesterday I met the boy is crying". However, in traditional word-level distillation, the student model only aligns the word-level probability distribution through KL divergence, resulting in the loss of source dependency structure knowledge, and the attention mechanism of its encoder fails to establish the key connection between "whom" and "met", resulting in the generation of "the boy I met yesterday is crying" during decoding, which destroys the structure of the original sentence's object clause.

[0003] This phenomenon exposes two key problems: first, word-level alignment based on KL divergence is inherently difficult to model cross-word syntactic relationships, resulting in the source encoder failing to effectively preserve structured knowledge such as dependency trees; second, the word-by-word optimization objective of traditional distillation deviates from the syntactic sensitivity required for machine translation, and when dealing with long-distance dependencies or nested structures (such as relative clauses, prepositional phrases), errors in the decoding process will be amplified step by step. The root cause is that existing methods lack an effective structured knowledge transfer mechanism, making it difficult for student models to faithfully maintain the source sentence syntactic framework during the decoding phase. In addition, the traditional encoder-decoder model compresses the entire source sentence into a fixed-length vector, which leads to information loss when processing long sentences. Although the Transformer architecture alleviates this problem through the self-attention mechanism, without explicitly modeling syntactic constraints, student models will still perform poorly when translating complex long sentences due to structural distortion. For example, when dealing with long sentences containing multiple modifiers, student models without syntactic constraints often make errors such as misplaced modifiers or confused central words. Therefore, how to organically integrate syntactic knowledge into the distillation process and build a source dependency structure preservation mechanism has become a key breakthrough for improving translation quality.

[0004] To address the above problems, many people have proposed several knowledge distillation methods that incorporate syntactic knowledge. Existing related methods can be mainly divided into two categories: one is the separate distillation framework, which first independently trains one or more teacher models that focus on syntactic tasks, and then transfers syntactic structure knowledge to the student model. For example, some methods integrate heterogeneous structure knowledge into a unified sequential LSTM encoder through knowledge distillation. The other category is explicit embedded distillation methods that directly incorporate syntactic information into the student model, usually using graph convolutional neural networks to learn sentence syntactic information and integrating it into the word embedding layer.

[0005] However, these methods have inherent drawbacks. First, the separate syntactic distillation framework requires additional training of a dedicated syntactic teacher model, which not only increases the complexity of the training process but may also lead to alignment issues between syntactic knowledge and translation tasks. Second, while methods that explicitly embed syntactic information can capture syntactic structural features, there is a mismatch between the rigid constraints of static syntactic trees and dynamic self-attention mechanisms. For example, word-syntactic node connections imposed by graph convolutional networks may interfere with the Transformer model's adaptive learning of long-distance dependencies in the self-attention mechanism. Furthermore, these methods add additional syntactic overhead (such as modules, linearization, representations, etc.) when introducing syntactic features, significantly increasing the model's complexity. A deeper problem lies in the fact that existing methods have failed to address the selective transfer of syntactic knowledge during the distillation process. Different samples contribute differently to the model during knowledge distillation, and the student model cannot achieve optimal results simply by learning all the knowledge from the teacher model. Research shows that distilled knowledge mainly comes from the teacher model's Top-1 predictions. Although existing methods have enhanced the learning of Top-1 information, the student model still struggles to fully learn the teacher model's syntactic knowledge, inevitably resulting in knowledge loss during the distillation process. Therefore, there is an urgent need for a distillation mechanism that can dynamically coordinate syntactic constraints and translation knowledge transfer, which can both preserve key dependency structures and avoid significantly increasing model complexity.

[0006] Based on the above background, a selective knowledge distillation method for neural machine translation based on dependency-constrained self-attention is proposed, which improves the performance of Transformer-based neural machine translation through a structured knowledge injection mechanism. Summary of the Invention

[0007] The technical problem to be solved by this invention is that it provides a selective knowledge distillation method for neural machine translation based on dependency-constrained self-attention, in order to solve the problem that translation models trained using knowledge distillation methods do not translate complex long sentences well. This invention improves the performance of machine translation.

[0008] The technical solution of this invention is: a selective knowledge distillation method for neural machine translation based on dependency-constrained self-attention, the method comprising:

[0009] Step 1: Syntactic Structure Analysis: Construct a syntax tree through dependency parsing to extract grammatical dependencies between words;

[0010] Step 2, Syntax Matrix Construction: Transform the syntax tree into a syntax matrix and establish a topological structure with explicit syntax constraints;

[0011] Step 3, Attention Enhancement Mechanism: The syntax matrix and attention weights are fused in the Transformer encoding layer to generate a syntax-guided attention distribution;

[0012] Step 4: Constructing the global cross-entropy queue: Establish a global queue to maintain historical cross-entropy values, and filter high-information-entropy words by sorting;

[0013] Step 5: Selective knowledge distillation: Selectively calculate attention KL divergence based on entropy values ​​to enhance the learning of key semantic regions;

[0014] Step 6, Model Training and Optimization: Combining soft-label cross-entropy and attention distillation loss, we achieve grammar-preserving model compression.

[0015] Furthermore, Step 1 includes:

[0016] Obtain the source text dependency tree: Given an input sentence X of length K, use the external parser Stanza to extract the dependency parser tree T = (V, E), where V represents the set of nodes in the dependency parser tree and E represents the set of edges in the dependency parser tree.

[0017] Furthermore, Step 2 includes:

[0018] Step 2.1: Based on the parsed dependency syntax tree, calculate word-level dependency distances, and traverse each word to calculate the dependency distance between words; define dependency distance as the path length from one word to another in the dependency syntax tree, and define the dependency distance between directly connected words as 1; and Dependency distance Represented as:

[0019]

[0020] in, express arrive The path, For indicator functions, and These are two nodes in the dependency syntax tree, and E represents the set of edges in the dependency syntax tree;

[0021] Step 2.2: Map discrete dependency distances to continuous constraints and construct a dependency scaling matrix. Its element calculation is expressed as:

[0022] in, Indicates the length of the input sentence. The Gaussian variance, set based on empirical values, is usually a very small number. It is used to reweight and align scores by quantifying the dependencies between words into Gaussian probability density values. As The The and the first Each element is actually a variance of . and input as Gaussian distribution; dependency scaling matrix Each line represents the syntactic dependency tightness of each word with other words.

[0023] Furthermore, Step 3 includes:

[0024] For the Layer encoders, self-attention mechanisms compute queries s and keys The alignment score is obtained by taking the dot product of the two numbers. The calculation formula is as follows:

[0025]

[0026] in, It is the dimension of the key, then a dependency scaling matrix is ​​used. Alignment score Perform weighted averaging to obtain the weighted alignment score. Then, the weighted alignment scores are normalized using the softmax function to obtain the scaled attention weights. Then, the outputs of all attention heads are concatenated, and a linear transformation is used to generate the final context vector with syntactic constraints. This is used for subsequent decoding, and the calculation formula is as follows:

[0027]

[0028]

[0029] in, Indicates the number of heads of attention. It is the first The value matrix of the layer encoder, Indicates the first The first layer encoder The output of each attention head, It is the first The output of the layer encoder is a linear transformation weight matrix.

[0030] Furthermore, Step 4 includes:

[0031] Cross-entropy global queue construction:

[0032] First, calculate the cross-entropy loss, given the source sentence. Its corresponding actual translation sentence For the first sentence in the target sentence The cross-entropy loss formula for each word is as follows:

[0033]

[0034] in, It is the size of the target vocabulary. It is an indicator function. The model parameters are: The conditional probability, For source sentences The nth word in For accurate sentence translation The m-th word in; Indicates the first sentence in the target sentence The real words in each position, Indicates the first sentence in the target sentence All words preceding that position, Represents a complete source language sentence;

[0035] The cross-entropy values ​​of the words in the current batch are added to the first-in-first-out (FIFO) global queue Q in sequence, and the cross-entropy of the earliest word that entered the queue is removed to maintain the size of the global queue. Then, all cross-entropy values ​​in the global queue are sorted and the ranking position of each word is calculated.

[0036] Furthermore, Step 5 includes:

[0037] KL distance is used to quantify the difference between the student model and the teacher model's predicted probability distribution for each word, i.e., using the conditional probability distribution given by the teacher model. Compare it with the conditional probability distribution predicted by the student model. By comparison, by minimizing the KL distance between the two, the student model can better learn from the teacher model how to predict the probability distribution of each word; knowledge distillation objective The calculation formula is as follows:

[0038]

[0039] in, and These are the parameter sets for the teacher model and the student model, respectively.

[0040] The knowledge distillation objective can be rewritten as follows:

[0041]

[0042] in, It is the conditional probability of the teacher model. The conditional probability distribution predicted by the student model. The one with the largest cross-entropy in batch A A collection of words, and It is the complementary part.

[0043] Furthermore, Step 6 includes:

[0044] The conventional training objective of neural machine translation models With knowledge distillation goals Joint optimization is performed, and the overall goal during training is to minimize the sum of the two objectives:

[0045]

[0046] Here, α is the weight that balances the two losses.

[0047] The present invention also provides a neural machine translation selective knowledge distillation system based on dependency-constrained self-attention, the system comprising: a module for performing the aforementioned neural machine translation selective knowledge distillation method based on dependency-constrained self-attention.

[0048] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the neural machine translation selective knowledge distillation method based on dependency-constrained self-attention.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned neural machine translation selective knowledge distillation method based on dependency-constrained self-attention.

[0050] The beneficial effects of this invention are:

[0051] 1. The neural machine translation selective knowledge distillation method based on dependency-constrained self-attention proposed in this invention performs excellently in translation tasks;

[0052] 2. The method of this invention demonstrates superior translation capabilities when processing sentences with complex structures and tight grammatical relationships. It is able to better capture the dependencies between semantics and syntax, thereby improving translation quality.

[0053] 3. When faced with low-resource language translation tasks, this invention can still maintain high translation quality, effectively alleviate the problem of data scarcity, and provide a feasible solution for machine translation in multilingual scenarios.

[0054] 4. The model compression strategy designed in this invention can effectively compress large models to a smaller size, significantly reduce the number of model parameters, and make them deployable on devices with extremely limited storage and computing resources, thus possessing good practicality.

[0055] 5. The method of this invention can not only reduce the model size and improve the inference speed, but also maintain high translation performance while ensuring lightweight design, thus achieving a good balance between performance and efficiency. Attached Figure Description

[0056] Figure 1 This is a framework diagram of the method in this invention;

[0057] Figure 2 This is a schematic diagram illustrating the effect of selective knowledge distillation in an embodiment of the present invention;

[0058] Figure 3 This refers to the BLEU scores of different sentence lengths on three datasets: English-German, English-French, and English-Vietnamese, in this embodiment of the invention.

[0059] Figure 4 The embodiments of the present invention are the original attention map, the dependency constraint matrix, and the attention map after linear combination. Detailed Implementation

[0060] Example 1: As Figure 1 As shown, a neural machine translation selective knowledge distillation method based on dependency-constrained self-attention is described, the method comprising:

[0061] Step 1: Syntactic Structure Analysis: Construct a syntax tree through dependency parsing to extract grammatical dependencies between words;

[0062] Furthermore, Step 1 includes:

[0063] Obtain the source text dependency tree: Given an input sentence X of length K, use the external parser Stanza to extract the dependency parser tree T = (V, E), where V represents the set of nodes in the dependency parser tree and E represents the set of edges in the dependency parser tree.

[0064] Step 2, Syntax Matrix Construction: Transform the syntax tree into a syntax matrix and establish a topological structure with explicit syntax constraints;

[0065] Furthermore, Step 2 includes:

[0066] Step 2.1: Based on the parsed dependency syntax tree, calculate word-level dependency distances, and traverse each word to calculate the dependency distance between words; define dependency distance as the path length from one word to another in the dependency syntax tree, and define the dependency distance between directly connected words as 1; and Dependency distance Represented as:

[0067]

[0068] in, express arrive The path, For indicator functions, and These are two nodes in the dependency syntax tree, and E represents the set of edges in the dependency syntax tree;

[0069] Step 2.2: Map discrete dependency distances to continuous constraints and construct a dependency scaling matrix. Its element calculation is expressed as:

[0070] in, Indicates the length of the input sentence. The Gaussian variance, set based on empirical values, is usually a very small number. It is used to reweight and align scores by quantifying the dependencies between words into Gaussian probability density values. As The The and the first Each element is actually a variance of . and input as Gaussian distribution; dependency scaling matrix Each line represents the syntactic dependency tightness of each word with other words.

[0071] Step 3, Attention Enhancement Mechanism: The syntax matrix and attention weights are fused in the Transformer encoding layer to generate a syntax-guided attention distribution;

[0072] Furthermore, Step 3 includes:

[0073] Syntactic Enhancement Attention Constraints: For the first Layer encoders, self-attention mechanisms compute queries s and keys The alignment score is obtained by taking the dot product of the two numbers. The calculation formula is as follows:

[0074]

[0075] in, It is the dimension of the key, then a dependency scaling matrix is ​​used. Alignment score Perform weighted averaging to obtain the weighted alignment score. Then, the weighted alignment scores are normalized using the softmax function to obtain the scaled attention weights. Then, the outputs of all attention heads are concatenated, and a linear transformation is used to generate the final context vector with syntactic constraints. This is used for subsequent decoding, and the calculation formula is as follows:

[0076]

[0077]

[0078] in, Indicates the number of heads of attention. It is the first The value matrix of the layer encoder, Indicates the first The first layer encoder The output of each attention head, It is the first The output of the layer encoder is a linear transformation weight matrix.

[0079] Step 4: Constructing the global cross-entropy queue: Establish a global queue to maintain historical cross-entropy values, and filter high-information-entropy words by sorting;

[0080] Furthermore, Step 4 includes:

[0081] Cross-entropy global queue construction:

[0082] First, calculate the cross-entropy loss, given the source sentence. Its corresponding actual translation sentence The goal of the neural machine translation model is to minimize the negative log-likelihood of the target word by calculating the cross-entropy loss at each position; for the first word in the target sentence... The cross-entropy loss formula for each word is as follows:

[0083]

[0084] in, It is the size of the target vocabulary. It is an indicator function. The model parameters are: The conditional probability, For source sentences The nth word in For accurate sentence translation The m-th word in; Indicates the first sentence in the target sentence The real words in each position, Indicates the first sentence in the target sentence All words preceding that position, Represents a complete source language sentence;

[0085] The cross-entropy values ​​of the words in the current batch are added to the first-in-first-out (FIFO) global queue Q in sequence, and the cross-entropy of the earliest word added to the queue is removed to maintain the size of the global queue. Then, all cross-entropy values ​​in the global queue are sorted and the ranking position of each word is calculated.

[0086] Step 5: Selective knowledge distillation: Selectively calculate attention KL divergence based on entropy values ​​to enhance the learning of key semantic regions;

[0087] Furthermore, Step 5 includes:

[0088] Knowledge distillation is performed by calculating the difference between the predicted probability distributions of the student model and the teacher model for each word. Typically, the Kullback-Leibler (KL) distance is used to quantify the difference between the predicted probability distributions of the student and teacher models for each word, i.e., utilizing the conditional probability distribution given by the teacher model. Compare it with the conditional probability distribution predicted by the student model. By comparison, by minimizing the KL distance between the two, the student model can better learn from the teacher model how to predict the probability distribution of each word; knowledge distillation objective The calculation formula is as follows:

[0089]

[0090] in, and These are the parameter sets for the teacher model and the student model, respectively.

[0091] The knowledge distillation objective can be rewritten as follows:

[0092]

[0093] in, It is the conditional probability of the teacher model. The conditional probability distribution predicted by the student model. The batch with the largest cross-entropy (Word CE) in batch A A collection of words, and It is the complementary part.

[0094] Step 6, Model Training and Optimization: Combining soft-label cross-entropy and attention distillation loss, we achieve grammar-preserving model compression.

[0095] Furthermore, Step 6 includes:

[0096] The conventional training objective of neural machine translation models With knowledge distillation goals Joint optimization is performed, and the overall goal during training is to minimize the sum of the two objectives:

[0097]

[0098] Here, α is the weight that balances the two losses.

[0099] The present invention also provides a neural machine translation selective knowledge distillation system based on dependency-constrained self-attention, the system comprising: a module for performing the aforementioned neural machine translation selective knowledge distillation method based on dependency-constrained self-attention.

[0100] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the neural machine translation selective knowledge distillation method based on dependency-constrained self-attention.

[0101] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned neural machine translation selective knowledge distillation method based on dependency-constrained self-attention.

[0102] To verify the effectiveness of the proposed model, the method model was compared with other state-of-the-art machine translation models, covering multiple datasets, including the WMT'14 English-German (WMT'14 EN-DE), IWLST'14 English-German (EN-DE), IWSLT'14 English-French (En-Fr), and IWSLT'15 English-Vietnamese (En-Vi) translation tasks. The comparison models include:

[0103] (1) Transformer: A basic model architecture built on the self-attention mechanism. It replaces the traditional recurrent and convolutional layers with a multi-head self-attention mechanism and a positional feedforward network, achieving efficient parallelization and effective learning of long-distance dependencies.

[0104] (2) Word-KD: A standard approach that focuses on the predicted probability distribution at the word level. The teacher model provides the probability distribution for each word as a soft label, and the student model improves its performance by learning these distributions.

[0105] (3) Seq-KD: This focuses on the overall generation of the sequence. The teacher model generates the distribution or output of the entire sequence, and the student model learns this sequence-level knowledge.

[0106] (4) Selective-KD: A selective knowledge distillation method based on Transformer, which guides student models to learn more valuable knowledge from teacher models effectively by designing selection strategies.

[0107] (5) TIE-KD: A knowledge distillation method that enhances Top-1 information by designing a hierarchical ranking loss to reinforce learning Top-1 information from the teacher model.

[0108] (6) CTCPMLM: A non-autoregressive translation method based on a pre-trained multilingual model. This method uses Connectionist Temporal Classification (CTC) loss to fine-tune the model and learns the representational ability of the teacher model by combining embedding distillation technology, thereby significantly improving translation quality while improving translation speed.

[0109] (7) GPT-3.5-turbo: A large language model whose core architecture is based on the Transformer decoder and employs a stacked self-attention mechanism and a feedforward neural network. Unlike the original Transformer, it only retains the decoder structure and achieves autoregressive generation through masked self-attention, supporting long text modeling and efficient parallel training.

[0110] Evaluation Metric: BLEU is used as the evaluation metric. A higher BLEU value indicates better performance.

[0111] The method of this invention was first compared with the WMT14 English-German benchmark dataset, and the results are shown in Table 2. The following conclusions were drawn: First, compared with the traditional methods Word-KD and Seq-KD, the method of this invention shows significant advantages in the WMT14 English-German translation task. Second, compared with several existing models trained using knowledge distillation methods, the method of this invention still maintains a leading performance. It is worth noting that although trained using a smaller Transformer (Base) architecture (6-layer encoder-decoder, 512-dimensional hidden states, 8 attention heads), the model performance surpasses that of a Transformer (Big) architecture with a larger number of parameters (6-layer encoder-decoder, 1024-dimensional hidden states, 16 attention heads). This indicates that the knowledge distillation strategy, which incorporates dependency syntactic information, not only maintains a smaller model size but also surpasses models with more parameters in translation performance, fully demonstrating the dual advantages of model compression and performance improvement.

[0112] It is particularly noteworthy that although the method of this invention lags behind the large language model GPT-3.5-turbo in terms of BLEU score, it has significant advantages in terms of training cost and inference speed. This advantage is of great significance in practical applications, especially for tasks with limited resources or high inference speed requirements.

[0113] Experimental results on the WMT'14 EN-DE public dataset demonstrate that the neural machine translation selective knowledge distillation method based on dependency-constrained self-attention exhibits high translation performance in translation tasks.

[0114] Table 1 shows a comparison of experimental results on the WMT'14 EN-DE dataset.

[0115] Furthermore, to verify the effectiveness of the proposed method, ablation experiments were conducted on the IWSLT'14 English-German, English-French, and IWSLT'15 English-Vietnamese translation tasks. The experiments aimed to explore the impact mechanism of dependency syntax constraints on performance during the knowledge distillation process of the teacher-student model. Three different scenarios for applying syntactic constraints were designed to analyze their impact on model performance: applying dependency syntax constraints only to the teacher model (Dep-Tea); applying dependency syntax constraints only to the student model (Dep-Stu); and applying dependency syntax constraints to both the teacher and student models (Dep-All). The experimental results are shown in Table 2.

[0116] Experimental results show that all experimental groups with dependency syntactic constraints outperform the student model (Baseline) using only selective knowledge distillation. In particular, when both the teacher and student models have syntactic constraints (Dep-All), this model achieves optimal performance on multiple translation tasks. This indicates that dependency syntactic constraints not only directly improve the syntactic representation ability of the student model but also enhance the syntactic information encoding process of the teacher model, making the knowledge transmitted by the teacher model more structured and instructive. Even when only syntactic constraints are applied to the student model (Dep-Stu), the model performance still shows a significant improvement, further demonstrating that syntactic constraints have a direct promoting effect on the learning process of the student model. Notably, when only constraints are applied to the teacher model (Dep-Tea), the performance is actually lower than the baseline model. This phenomenon indicates that if the student model lacks corresponding syntactic constraint support, its syntactic representation ability is insufficient to decode the syntactic enhancement knowledge of the teacher model, leading to a knowledge transfer mismatch during the distillation process—the syntactic-related knowledge output by the teacher model is converted into noise due to the insufficient decoding ability of the student model, ultimately affecting the overall performance.

[0117] Table 2 shows a comparison of experimental results on the IWSLT'14 dataset.

[0118] Furthermore, to evaluate the impact of selective knowledge distillation methods on student model learning of teacher model knowledge, experiments were conducted on the IWSLT'14 English-German dataset, using Transformer as the baseline model and combining multiple distillation methods. During the experiments, the BLEU score of each training batch on the validation set was recorded. Figure 2 The relevant results are presented in the document.

[0119] Experimental results show that selective knowledge distillation, by selecting knowledge from the teacher model that is helpful to student learning, avoids interference from irrelevant or redundant information, enabling the student model to extract valuable knowledge more effectively from the teacher model. Simultaneously, structural alignment distillation combined with syntactic information further enhances the transfer of cross-linguistic knowledge, achieving optimal performance for the student model in translation tasks. Therefore, selective knowledge distillation not only improves model training efficiency but also directly improves translation quality, demonstrating the effectiveness of this method in enhancing neural machine translation performance.

[0120] To further verify that this invention has better translation performance for complex long sentences, experiments were conducted on sentences of different lengths. The experimental results are as follows: Figure 3 As shown in the figure. Experimental results demonstrate that the method of this invention outperforms the comparison model across all sentence length ranges, with a particularly significant advantage in long sentence translation. Specifically, traditional self-attention mechanisms struggle to capture long-distance dependencies due to the sparsity of attention in long sequences, while the method of this invention effectively alleviates this problem by adjusting attention weights through dependency syntax constraints, thereby strengthening the focus on key dependency structures. In high-resource language pairs such as English-German and English-French, the performance advantage of the method of this invention gradually expands with increasing sentence length, especially in the translation of long sentences exceeding 40 words, where the BLEU score is significantly higher than other models.

[0121] To explore the impact mechanism of model structural simplification on knowledge distillation performance, student models with different capacity configurations were systematically constructed for experiments. As shown in Table 3, the experimental results demonstrate that the method of this invention not only reduces model size and improves inference speed, but also maintains high translation performance while preserving lightweight design, achieving a good balance between performance and efficiency.

[0122] Table 3 shows a comparison of experimental results on the IWSLT'14 English-German dataset.

[0123] To verify the guiding effect of dependency syntactic constraints on the self-attention mechanism, a visual analysis of the self-attention network injected with dependency information was carried out. A visual analysis of the attention distribution was carried out on the complex long sentence "When each of you in this room were born, there were 6,000 languages spoken on the planet." (Chinese translation: "When everyone here was born, there were 6,000 languages being used on this planet.") The experimental results are as Figure 4 shown. The experimental results show that by introducing dependency constraints, the method of the present invention makes the attention distribution of the model more structural, can more accurately focus on the syntactic backbone components, and improves the modeling ability for long-distance dependency relationships. Especially when dealing with complex modification structures, explicit syntactic constraints effectively strengthen the model's attention to core semantic units, thereby optimizing the attention distribution pattern. This finding further verifies the importance of dependency information in the self-attention mechanism, indicating that combining syntactic knowledge can enhance the model's structural perception ability and improve the modeling quality of long sentences and complex sentences.

[0124] Example 2: As Figure 1 shown, a selective knowledge distillation method for neural machine translation based on dependency-constrained self-attention includes the following steps:

[0125] Step1. Given a source language sentence, use the external syntactic parser Stanza to extract the dependency syntactic tree of the sentence.

[0126] Step2.1. Based on the parsed dependency tree, calculate the dependency distance at the lexical level, and traverse each word to calculate the dependency distance between words.

[0127] Step2.2. Through the calculated dependency distance, map the discrete distance to a continuous constraint, and construct a dependency scaling matrix through a Gaussian function . Each row of the dependency scaling matrix represents the syntactic dependency tightness of each word with other words.

[0128] Step3. Attention enhancement mechanism: In the self-attention mechanism of the Transformer encoder, linearly combine the dependency adjacency matrix with the original attention weights to dynamically generate the attention distribution under syntactic constraints, thereby guiding the model to focus on the syntactic structure;

[0129] Step 4: Construction of the global cross-entropy queue: Based on the word-level cross-entropy (Word-CE) predicted by the teacher model, a first-in-first-out global queue is constructed. The cross-entropy values ​​of the current batch of words are added to the queue in sequence. The oldest historical value in the queue is removed to maintain the stability of the capacity. At the same time, the cross-entropy values ​​in the queue are sorted and the ranking position of each word is calculated to provide a basis for subsequent distillation and screening.

[0130] Step 5, Selective Knowledge Distillation: Using the constructed cross-entropy global queue, based on the sorting results, select the higher cross-entropy and calculate the KL divergence loss of the attention distribution between the teacher model and the student model, thereby strengthening the student model's learning of regions with higher information entropy and richer semantics.

[0131] Step 6, Model Training and Optimization: By jointly optimizing the soft-label cross-entropy loss and selective attention distillation loss, the student model retains more structural grammatical knowledge while compressing, thereby improving its translation quality and syntactic preservation ability.

[0132] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A neural machine translation selective knowledge distillation method based on dependency constraint self-attention, characterized in that: The method comprises: Step1, syntactic structure analysis: constructing a syntactic tree through dependency syntactic analysis, and extracting the syntactic dependency relationship between words; Step2, syntactic matrix construction: converting the syntactic tree into a syntactic matrix to establish the topological structure of explicit syntactic constraints; Step3, attention enhancement mechanism: integrating the syntactic matrix and attention weight in the Transformer encoding layer to generate a grammar-guided attention distribution; Step4, cross-entropy global queue construction: establishing a global queue to maintain historical cross-entropy values, and selecting high information entropy words through sorting; Step5, selective knowledge distillation: selectively calculating attention KL divergence according to entropy value sorting to strengthen the learning of key semantic areas; Step6, model training and optimization: combining soft label cross-entropy and attention distillation loss to realize model compression with grammar preservation.

2. The neural machine translation selective knowledge distillation method based on dependency constraint self-attention according to claim 1, wherein: The Step1 comprises: Obtaining a source text dependency tree: given an input sentence X with length K, use an external syntax parser Stanza to extract the dependency syntax tree T = (V, E) of the sentence, where V represents the node set in the dependency syntax tree, and E represents the edge set in the dependency syntax tree.

3. The neural machine translation selective knowledge distillation method based on dependency constraint self-attention according to claim 1, wherein: The Step2 comprises: Step 2.1, based on the parsed dependency syntax tree, the dependency distance of word level is calculated, and each word is traversed to calculate the dependency distance between words; the dependency distance is defined as the path length traversed on the dependency syntax tree from one word to another word, and the dependency distance between directly connected words is defined as 1; the dependency distance of words and is represented as: ​ ； wherein, denotes to a path, is an indicator function, and are two nodes in a dependency parse tree, E denotes a set of edges in the dependency parse tree; Step 2.2, mapping discrete dependency distances to continuous constraints, constructing dependency scaling matrix The element calculation is expressed as: ; wherein, denotes the length of the input sentence, a Gaussian variance set according to empirical values, typically a small number, used to reweight the alignment scores by quantifying the dependency proximity between words as a Gaussian probability density value; as the st and nd elements, is actually a Gaussian distribution with variance and input . Each row of the dependency scaling matrix represents the syntactic dependency tightness of each word with other words.

4. The neural machine translation selective knowledge distillation method based on dependency constraint self-attention according to claim 1, wherein: The Step3 comprises: For the first layer encoder, the self-attention mechanism gets alignment scores by computing the dot product of query and key , with the formula as follows: ； wherein, is the dimension of the key, after using the dependency scaling matrix aligning the scores weighting, to obtain the weighted alignment scores , and normalizing the weighted alignment scores by a softmax function to obtain the scaled attention weights , and concatenating the outputs of all attention heads and generating the final context vector with syntactic constraints through linear transformation for subsequent decoding, the calculation formula is as follows: ； ； in, Indicates the number of heads of attention. It is the first The value matrix of the layer encoder, Indicates the first The first layer encoder The output of each attention head, It is the first The output of the layer encoder is a linear transformation weight matrix.

5. The neural machine translation selective knowledge distillation method based on dependency constraint self-attention according to claim 1, wherein: The Step4 comprises: Cross-entropy global queue construction: First, calculate the cross-entropy loss, given the source sentence. Its corresponding actual translation sentence For the first sentence in the target sentence The cross-entropy loss formula for each word is as follows: ； in, It is the size of the target vocabulary. It is an indicator function. The model parameters are: The conditional probability, For source sentences The nth word in For accurate sentence translation The m-th word in; Indicates the first sentence in the target sentence The real words in each position, Indicates the first sentence in the target sentence All words preceding that position, Represents a complete source language sentence; The cross-entropy values of the current batch of words are sequentially added to the first-in-first-out FIFO global queue Q, and the word cross-entropy that entered the queue earliest is removed to maintain the global queue capacity size; then, all cross-entropy values in the global queue are sorted to calculate the ranking position of each word.

6. The neural machine translation selective knowledge distillation method based on dependency constraint self-attention according to claim 1, wherein: The Step5 comprises: The KL distance is used to quantify the difference between the predicted probability distribution of each word by the student model and the teacher model, i.e. the conditional probability distribution given by the teacher model , which is compared with the conditional probability distribution predicted by the student model . By minimizing the KL distance between the two, the student model can better learn from the teacher model how to predict the probability distribution of each word; the knowledge distillation objective The calculation formula is as follows: ； wherein, and are parameter sets of the teacher model and the student model, respectively; The knowledge distillation target is re-expressed as the following formula: ； in, It is the conditional probability of the teacher model. The conditional probability distribution predicted by the student model. The one with the largest cross-entropy in batch A A collection of words, and It is the complementary part.

7. The neural machine translation selective knowledge distillation method based on dependency constraint self-attention according to claim 1, wherein: The Step6 comprises: Conventional training objectives for neural machine translation models with a knowledge distillation objective Joint optimization is performed, with the objective during the overall training process being to minimize the sum of the two objectives: ； Wherein, α is the weight of balancing the two losses.

8. A neural machine translation selective knowledge distillation system based on dependency constrained self-attention, characterized in that, The system comprises a module for executing the dependency constraint self-attention based neural machine translation selective knowledge distillation method according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the dependency constraint self-attention based neural machine translation selective knowledge distillation method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the dependency constraint self-attention based neural machine translation selective knowledge distillation method according to any one of claims 1 to 7.

Images