Semi-supervised machine translation method and system based on multi-temperature self-distillation and lightweight cooperation

Abstract

The invention discloses a semi-supervised machine translation method and system based on multi-temperature self-distillation and lightweight collaboration, and the method comprises the steps: converting a text sequence in a training set into a lexical sequence, and inputting the lexical sequence into a teacher model after the position of the lexical sequence is coded; executing multi-temperature knowledge distillation in the teacher model to generate a multi-temperature soft label; the teacher model and the student model extract lightweight intermediate characterization through a grouping query attention mechanism; the teacher model and the student model are aligned; the student model takes the multi-temperature soft label and the lightweight intermediate representation as supervision signals, and updates own parameters through joint loss; inputting a to-be-translated source statement, encoding the source statement into a sub-word sequence by using a word segmentation device, and mapping the sub-word sequence into a word list ID sequence; filling the word list ID sequence to a fixed length, then sending the word list ID sequence into the student model with knowledge distillation completed, and outputting target language probability distribution; and decoding and textualizing the target language probability distribution to obtain a target language text. According to the invention, collaborative optimization of light weight and precision enhancement is effectively realized.

Description

Technical Field

[0001] This invention relates to natural language processing technology, specifically to a semi-supervised machine translation method and system based on multi-temperature self-distillation and lightweight collaboration. Background Technology

[0002] Neural Machine Translation (NMT), as one of the core tasks of Natural Language Processing, plays a crucial role in cross-language communication and global information exchange. Models like the Transformer, with their self-attention mechanism, have significantly improved translation quality. However, their massive number of parameters (e.g., the Base model has approximately 110M parameters) and high computational complexity (e.g., a single inference requires billions of floating-point operations) severely limit their deployment in resource-constrained scenarios (such as mobile devices and low-computing-power language communities). Traditional lightweight methods compress model size through model pruning, quantization, or architectural simplification (e.g., TinyBERT), but this often comes at the cost of translation accuracy, with significant performance degradation, especially in low-resource language pairs or complex semantic scenarios.

[0003] In existing technologies, knowledge distillation (KD) offers a potential path to mitigate the performance loss of lightweight models. For example, the paper "LLaVA-MoD: Making LLaVATiny via MoE Knowledge Distillation", ICLR 2025, proposes a method to optimize lightweight multimodal large models through a hybrid expert (MoE) architecture and knowledge distillation. This method, using only 0.3% of the data and 23% of the activation parameters, enables a 2B small model to outperform a 7B large model by 8.8% in overall performance and surpasses the teacher model in the hallucination detection task. However, such methods rely on the presence of a high-quality teacher model, and training and deploying dual models under low-resource conditions requires twice the computational resources, limiting their practicality. Furthermore, the paper "TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillation", arXiv preprint arXiv:2502.15016, 2025, proposes a cross-architecture knowledge distillation framework. This framework uses an MLP as the student model and other complex, advanced architectures (such as Transformer and CNN) as the teacher model. By distilling the advantages of complex models into lightweight models, it significantly reduces computational burden while substantially improving prediction accuracy. However, its strict layer matching mechanism is difficult to adapt to the layer reduction requirements of lightweight architectures.

[0004] Recently, self-distillation has gained attention as an emerging technique that does not require external teachers. For example, the paper "Dual teachers for self-knowledge distillation" (PatternRecognition, 2024) proposes a dual-teacher self-distillation method that significantly improves model performance and reduces computational costs by utilizing the model's own intermediate layer predictions or historical predictions for knowledge distillation. However, directly applying such methods to NMT faces two major challenges:

[0005] 1. High-dimensional sparsity of sequence generation: The output space dimension of translation tasks is much higher than that of classification tasks, and traditional self-distilled soft labels are easily affected by noise;

[0006] 2. Lightweight model capacity bottleneck: Parameter compression leads to insufficient model representation ability, making it difficult to simultaneously optimize task loss and self-distillation constraint.

[0007] To address the aforementioned issues, existing research, such as the paper "Self-Evolution Knowledge Distillation for LLM-based Machine Translation" (arXiv preprint arXiv:2412.15303,2024), attempts to introduce self-distillation into NMT. ​​However, it is not optimized for lightweight architectures and exhibits poor training stability in low-resource scenarios. Furthermore, while lightweight model designs (such as TinyBERT) reduce computational overhead through layer reduction and dimensionality compression, they lack dynamic knowledge transfer mechanisms, leading to a significant decline in translation quality for long-tail words and complex sentence structures. Summary of the Invention

[0008] This invention addresses the problems of insufficient accuracy and high computational cost of current lightweight neural machine translation models in resource-constrained scenarios. It provides a semi-supervised machine translation method and system based on multi-temperature self-distillation and lightweight collaboration. By designing a knowledge transfer mechanism and model compression strategy in stages, it overcomes the difficulties of low knowledge transfer efficiency and excessive memory consumption in traditional methods.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy includes the following steps:

[0011] Step 1: Collect text sequences to form a training set, convert the text sequences in the training set into word sequences through a word embedding layer, and add positional encoding to the word sequences;

[0012] Step 2: Input the position-encoded lexical sequence into the teacher model; perform knowledge distillation between the teacher and student models; the knowledge distillation process includes: performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels; both the teacher and student models extract lightweight intermediate representations through a grouped query attention mechanism; project the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 onto the same dimension after layer normalization, so that the teacher and student models are aligned; the student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss to complete knowledge distillation;

[0013] Step 3: Input the source sentence to be translated, use a word segmenter to encode the source sentence into a sequence of sub-words, and map it to a sequence of vocabulary IDs;

[0014] Step 4: After filling the vocabulary ID sequence to a fixed length, feed it into the student model that has completed knowledge distillation, and output the target language probability distribution;

[0015] Step 5: Decode and textify the probability distribution of the target language to obtain the target language text.

[0016] To optimize the above technical solution, the specific measures also include:

[0017] Further, in step 1, adding positional encoding to the word sequence specifically involves:

[0018] Position encoding is generated using sine and cosine functions:

[0019]

[0020] Where pos is the position index, i is the dimension index, and d is the position index. model For the embedding dimension, PE (pos,2i) and PE (pos,2i+1) These represent the sine and cosine position codes of position pos in dimensions 2i and 2i+1, respectively, which are used to provide the teacher model with the position information of words in the word sequence.

[0021] Furthermore, in step 2, the step of performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels specifically involves:

[0022] In the first 2000 steps of training, the current teacher model parameters are saved every 100 steps as an instantaneous teacher, retaining only the output probability distribution and intermediate representations for the current training step. The probability distribution is calculated using the softmax function and used as a soft label, as shown in the following formula:

[0023] T t=4.0-0.0015t(0≤t≤2000);

[0024] In the formula, x represents the tensor composed of the word sequence encoded by the position of the input at the current training step, and P t (x) represents the teacher model's processing of input x at temperature T after training steps t. t The softened output probability distribution, i.e., the soft labels used for knowledge distillation; softmax(·) denotes the softmax function, z t (x) represents the teacher model's original output to input x at training step t, i.e., the vocabulary dimension real vector logits, where each dimension corresponds to the unnormalized prediction score of the corresponding word in the vocabulary; T t It is the temperature coefficient, when T t When T = 4, it is defined as high temperature. The probability distribution of the output at this time is defined as the high temperature soft label. t When T = 2.5, it is defined as medium temperature. The probability distribution of the output at this time is defined as a medium-temperature soft label. t When = 1, it is defined as low temperature, and the output probability distribution is defined as low temperature soft label.

[0025] Furthermore, in step 2, both the teacher model and the student model extract lightweight intermediate representations through a grouped query attention mechanism, specifically as follows:

[0026] Both the teacher and student models execute a grouped query attention mechanism using the following formula to extract lightweight intermediate representations:

[0027] H gqa =GQA(Q,K,V)=concat(head1,...,head j ,...,head n W O

[0028] In the formula, H gqa It is a lightweight intermediate representation. GQA stands for Grouped Query Attention mechanism, which divides the query header into n groups and shares the key K and value V in each group. Q is the query vector, which represents the query direction of the teacher model and the student model on the input data. K is the key vector, which represents the features of the input data. V is the value vector, which represents the actual content of the input data.

[0029] head j W represents the attention weighted vector generated by the j-th query header, where concat represents the concatenation operation. O It is a projection matrix used to project the concatenated vectors onto the target dimension, generating the final lightweight intermediate representation H. gqa ;

[0030] The formula for calculating the attention weighted vector of the j-th query header is as follows:

[0031]

[0032] In the formula, softmax(·) represents the softmax function, and Q j This is the j-th query header, where K and V refer to the shared key and value of the query headers, respectively. The superscript T indicates transpose, and d is the dimension of the key vector. It is a scaling factor used to prevent gradient vanishing.

[0033] Further, in step 2, the step of projecting the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 onto the same dimension after layer normalization, so that the teacher model and the student model are aligned, specifically involves:

[0034] First, lightweight intermediate representations of the l-th layer of the student model. With student model weights W s Multiplication, lightweighting the l+1th intermediate representation of the teacher model. Teacher model weights W t Multiply them, and then perform layer normalization separately to map them to the same feature space;

[0035] Calculate the local distillation loss, which is only used to update the student model weights W. s and subsequent layer parameters of the student model, and weights W of the teacher model. t The formula for partial distillation loss while maintaining freezing is as follows:

[0036]

[0037] In the formula, L local This represents the local distillation loss, used to measure the difference between the student model and the teacher model at a certain layer output. D represents the dimension of the feature, i.e. the number of elements in each lightweight intermediate representation.

[0038] Furthermore, in step 2, the student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss as follows:

[0039] The student model uses the high-temperature soft label, medium-temperature soft label, low-temperature soft label, and lightweight intermediate representation H generated by the teacher model. gqa As a monitoring signal, its own parameters are updated through a joint loss; the joint loss is the sum of the local distillation loss and the task cross-entropy loss.

[0040] Furthermore, the method also includes:

[0041] During the training phase, in each forward propagation, the activation value of each neuron is calculated, and the top 50% of neurons in terms of activation value are selected. Gradient updates are performed only on these activated neurons.

[0042] During the inference phase, in each forward propagation, the activation value of each neuron is calculated, and the top 30% of neurons in terms of activation value are selected. Only the selected neurons are used to calculate the sparse activation hidden state of the current layer, and then immediately sent to the next layer decoder to continue sequence generation until the translation is output.

[0043] Furthermore, step 4 specifically involves:

[0044] After filling the vocabulary ID sequence to a fixed length, it is fed into the student model that has completed knowledge distillation, and forward computation is performed. The last layer outputs a logits vector sequence. The softmax function with a temperature coefficient of 1 is applied to the logits vector sequence to obtain the target language probability distribution.

[0045] Furthermore, step 5 specifically includes:

[0046] Perform Beam Search decoding on the target language probability distribution, setting the beam width beam=4 and the length penalty lengthpenalty=1;

[0047] Search for the candidate translation ID sequence with the highest score within the maximum length limit, and perform BPE merge character restoration, subtotalization, and punctuation normalization operations on the highest-scoring translation ID sequence to obtain the target language text.

[0048] This invention also proposes a semi-supervised machine translation system based on multi-temperature self-distillation and lightweight synergy, comprising:

[0049] The training set creation module is used to collect text sequences to form a training set. It converts the text sequences in the training set into word sequences through a word embedding layer and adds positional encoding to the word sequences.

[0050] The knowledge distillation module is used to input the position-encoded lexical sequence into the teacher model; knowledge distillation is performed between the teacher model and the student model; the knowledge distillation process includes: performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels; both the teacher model and the student model extract lightweight intermediate representations through a grouped query attention mechanism; the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 are projected onto the same dimension after layer normalization, so that the teacher model and the student model are aligned; the student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss to complete the knowledge distillation.

[0051] The target language generation module receives the source sentence to be translated, encodes the source sentence into a sequence of sub-words using a word segmenter, and maps it to a sequence of vocabulary IDs; after padding the vocabulary ID sequence to a fixed length, it is fed into the student model that has completed knowledge distillation, and outputs the target language probability distribution; the target language probability distribution is decoded and textualized to obtain the target language text;

[0052] The physical output module is used to output the target language in text or speech form.

[0053] The beneficial effects of this invention are: (1) a dual breakthrough in knowledge transfer and performance. Through the dynamic three-temperature knowledge distillation strategy (4.0→1.0 linear decay), the model expands the solution space exploration in the early stage of training with the high temperature segment, balances global and local features in the middle stage with the medium temperature segment, and accurately optimizes semantics in the later stage with the low temperature segment, so that the BLEU value of the WMT2024 English and German test set reaches 28.7 (1.4 higher than the fixed temperature), while reducing redundant calculations by 15%-20% and controlling the loss fluctuation within ±0.02. The high-temperature segment fuzzy transfer mechanism further improves the noise robustness by 23%, solves the pain point of traditional distillation requiring manual temperature adjustment, and realizes the synergistic optimization of accuracy and efficiency in resource-constrained scenarios.

[0054] (2) Lightweight architecture and comprehensive upgrade in applicability. Adopting GQA grouped query attention (4 query heads sharing 1 key-value head) and a cross-layer parameter sharing strategy, the computational cost of the key-value matrix is ​​reduced by 75%, and parameter storage is reduced to 28% of the original model. Combined with dynamic sparse inference (retaining 30% of highly activated neurons), the actual computational cost is reduced to 35%, inference speed is increased by 22%, and GPU memory usage is reduced by 40%, supporting deployment on devices with 4GB of GPU memory. This design balances lightweight design with high performance, expanding the application potential of low-resource scenarios such as mobile devices and edge computing, and providing an efficient solution for real-time translation and offline neural machine translation. Attached Figure Description

[0055] Figure 1 This is a diagram of the semi-supervised machine translation framework proposed in this invention, based on multi-temperature self-distillation and lightweight synergy. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] Example 1

[0058] This invention proposes a semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy, such as... Figure 1 As shown, it includes the following steps:

[0059] Step 1: Collect text sequences to form a training set. Convert the text sequences in the training set into word sequences using a word embedding layer, and add positional encoding to the word sequences. Specifically, adding positional encoding to the word sequences involves:

[0060] Position encoding is generated using sine and cosine functions:

[0061]

[0062] Where pos is the position index, i is the dimension index, and d is the position index. model For the embedding dimension, PE (pos,2i) and PE (pos,2i+1) These represent the sine and cosine position codes of position pos in dimensions 2i and 2i+1, respectively, which are used to provide the teacher model with the position information of words in the word sequence.

[0063] In this embodiment, the teacher model uses a 12-layer Transformer structure with a hidden layer dimension of 512, a feedforward network dimension of 2048, and 8 attention heads, for a total of approximately 110M parameters. The student model uses a 6-layer Transformer structure with a hidden layer dimension of 384, a feedforward network dimension of 1536, and 4 attention heads. The decoder shares parameters between layers 1 and 3, and between layers 2 and 4, compressing the total number of parameters to 78M, a reduction of approximately 29% compared to the teacher model.

[0064] Step 2: Input the position-encoded lexical sequence into the teacher model; perform knowledge distillation between the teacher and student models; the knowledge distillation process includes: performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels; both the teacher and student models extract lightweight intermediate representations through a grouped query attention mechanism; project the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 onto the same dimension after layer normalization, so that the teacher and student models are aligned; the student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss to complete the knowledge distillation.

[0065] In the teacher model, multi-temperature knowledge distillation is performed to generate multi-temperature soft labels, specifically as follows:

[0066] In the first 2000 steps of training, the current teacher model parameters are saved every 100 steps as an instantaneous teacher, retaining only the output probability distribution and intermediate representations for the current training step. The probability distribution is calculated using the softmax function and used as a soft label, as shown in the following formula:

[0067] T t =4.0-0.0015t(0≤t≤2000);

[0068] In the formula, x represents the tensor composed of the word sequence encoded by the position of the input at the current training step, and P t (x) represents the teacher model's processing of input x at temperature T after training steps t. t The softened output probability distribution, i.e., the soft labels used for knowledge distillation; softmax(·) denotes the softmax function, z t (x) represents the teacher model's original output to input x at training step t, i.e., the vocabulary dimension real vector logits, where each dimension corresponds to the unnormalized prediction score of the corresponding word in the vocabulary; T t It is the temperature coefficient, which controls the smoothness of the output distribution. When T t When T = 4, it is defined as high temperature. The probability distribution of the output at this time is defined as the high temperature soft label. t When T = 2.5, it is defined as medium temperature. The probability distribution of the output at this time is defined as a medium-temperature soft label. t When the value is 1, it is defined as low temperature, and the output probability distribution at this time is defined as low temperature soft label. High temperature produces the smoothest distribution, which can amplify the signal of low probability words and help students capture long-distance semantic associations; medium temperature achieves a balance between category distinction and diversity, preventing the model from converging too early to the high-frequency word shortcut; low temperature is close to hard label, which can provide sharp supervision in the later stage of training and accurately preserve local semantic details. The simultaneous output of three temperature soft labels allows students to absorb the knowledge of the "global-intermediate-local" layers in a single forward pass, and the linear decay mechanism ensures a smooth transition without additional computational cost or manual parameter tuning. Therefore, only the above three discrete temperatures are needed to cover the smoothness spectrum required for the entire training cycle.

[0069] When temperature T t At higher levels, such as T t =4.0, the output distribution of the softmax function becomes smoother. This means the probability differences between categories decrease, i.e., the entropy of the output distribution increases. This smooth distribution helps the model capture the global features of the data in the early stages of training, because the model does not overly rely on specific categories (especially high-frequency words), but focuses more on learning the overall distribution and features of the data. When temperature T t At lower levels, such as T t When T = 1.0, the output distribution of the softmax function becomes sharper. This means that high-probability classes have higher probabilities, and low-probability classes have lower probabilities, i.e., the entropy of the output distribution decreases. This sharp distribution helps the model retain more semantic details in the later stages of training because the model can more accurately reflect the local features of the data and the differences between classes. According to equation T... t= 4.0 - 0.0015t (0 ≤ t ≤ 2000), we can see that the temperature T t The temperature decreases linearly with time step t. This means that in the first 2000 steps of training, the temperature gradually decreases from 4.0 to 1.0. This design conforms to the general pattern of model training: in the early stages of training, a higher temperature is used to soften the output distribution to help the model capture the global features of the data; in the later stages of training, a lower temperature is used to preserve the details of the output distribution to optimize the model's learning of the local semantics of the data. Therefore, in the initial stage of model training, because the model parameters fluctuate greatly, using a higher temperature can soften the output distribution and help the model capture the global features of the data; in the later stages of model training, because the model parameters tend to stabilize, using a lower temperature can preserve the details of the output distribution and optimize the model's learning of the local semantics of the data. This strategy of dynamically adjusting the temperature can balance the learning of global features and local semantics at different stages of model training, thereby improving the final performance of the model.

[0070] To address the high computational complexity of standard multi-head attention, which makes it difficult to apply in lightweight models, this application proposes a grouped query attention mechanism. This mechanism allows query heads to independently learn different semantic perspectives, while key-value heads are shared to reduce the number of parameters. For example, when translating "The cat sat on the mat," different query heads can focus on semantic roles such as "cat" (subject), "sat" (action), and "mat" (object), while the key-value head provides a unified contextual representation.

[0071] The specific process by which both the teacher and student models extract lightweight intermediate representations through the grouped query attention mechanism is as follows:

[0072] Both the teacher and student models execute a grouped query attention mechanism using the following formula to extract lightweight intermediate representations:

[0073] H gqa =GQA(Q,K,V)=concat(head1,...,head j ,...,head n W O

[0074] In the formula, H gqa It is a lightweight intermediate representation. GQA stands for Grouped Query Attention mechanism, which divides the query header into n groups and shares the key K and value V in each group. Q is the query vector, which represents the query direction of the teacher model and the student model on the input data. K is the key vector, which represents the features of the input data. V is the value vector, which represents the actual content of the input data.

[0075] head jW represents the attention weighted vector generated by the j-th query header, where concat represents the concatenation operation. O It is a projection matrix used to project the concatenated vectors onto the target dimension, generating the final lightweight intermediate representation H. gqa ;

[0076] The formula for calculating the attention weighted vector of the j-th query header is as follows:

[0077]

[0078] In the formula, softmax(·) represents the softmax function, and Q j This is the j-th query header, where K and V refer to the shared key and value of the query headers, respectively. The superscript T indicates transpose, and d is the dimension of the key vector. It is a scaling factor used to prevent gradient vanishing.

[0079] GQA is an optimized attention mechanism that reduces computation and parameter count by grouping query heads and sharing keys and values ​​within each group, while preserving the semantic diversity of multi-head attention mechanisms. This mechanism is particularly effective in large models because it can significantly reduce computational resource consumption while maintaining model performance. The head configuration is: 4 query heads sharing 1 set of key-value heads. Below is a comparison of computational complexity:

[0080] Attention Type computational complexity Standard long position (8 heads) O(8N 2 D)]]> GQA-4 (4 query headers) <![CDATA[O(4N 2 D+N 2 )]]>

[0081] In the knowledge transfer process, the intermediate layer representations of the teacher model often contain rich semantic information. However, directly passing these features to the student model may lead to the loss of semantic information because the structure of the student model may differ from that of the teacher model. To alleviate this problem, this application adopts an intermediate representation elastic alignment method, which projects the lightweight intermediate representations of the l-th layer of the student model and the (l+1)-th layer of the teacher model onto the same dimension after layer normalization, thereby aligning the teacher and student models and solving the problem of semantic information loss in knowledge transfer caused by differences in the number of layers and dimensions. The specific process is as follows:

[0082] First, lightweight intermediate representations of the l-th layer of the student model. With student model weights W s Multiplication, lightweighting the l+1th intermediate representation of the teacher model. Teacher model weights W t The two samples are multiplied and then layer normalized separately to map them to the same feature space. Layer normalization is a normalization technique that normalizes the features of each sample to a mean of 0 and a variance of 1, thereby reducing internal covariate bias.

[0083] Calculate the local distillation loss, which is only used to update the student model weights W. s and subsequent layer parameters of the student model, and weights W of the teacher model. t By maintaining the freeze state, the student model is forced to reproduce the teacher's high-level semantics without increasing the teacher's inference overhead, thus achieving inter-layer alignment and compensating for the knowledge gaps caused by the reduction in the number of student models. The formula for the local distillation loss is as follows:

[0084]

[0085] In the formula, L local This represents the local distillation loss, used to measure the difference between the student model and the teacher model at a certain layer output. D represents the dimension of the feature, i.e. the number of elements in each lightweight intermediate representation.

[0086] The student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss as follows:

[0087] The student model uses the high-temperature soft label, medium-temperature soft label, low-temperature soft label, and lightweight intermediate representation H generated by the teacher model. gqa As a monitoring signal, its own parameters are updated through a joint loss; the joint loss is the sum of the local distillation loss and the task cross-entropy loss.

[0088] Step 3: The user inputs the source sentence to be translated via keyboard, file handle or mobile API, for example: “The transformer architecture is powerful.” The tokenizer encodes the source sentence into a sub-word sequence ['The','trans','former','arch','itecture','is','powerful','.'] and maps it to a vocabulary ID sequence [812,3341,2205,1823,2884,83,1621,4].

[0089] Step 4: After filling the vocabulary ID sequence to a fixed length, feed it into the student model that has completed knowledge distillation and output the target language probability distribution. Specifically, after filling the vocabulary ID sequence to a fixed length, feed it into the student model that has completed knowledge distillation, perform forward computation, and output the logits vector sequence in the last layer. Apply the softmax function with a temperature coefficient of 1 to the logits vector sequence to obtain the target language probability distribution.

[0090] Step 5: Decode and transcribe the target language probability distribution to obtain the target language text. Specifically:

[0091] Perform Beam Search decoding on the target language probability distribution, setting the beam width beam=4 and the length penalty lengthpenalty=1;

[0092] Search for the highest-scoring candidate translation ID sequence ['Die','Trans','former','-Arch','itektur','ist',] within the maximum length limit. '.'], perform BPE merge character restoration, subtotalization, and punctuation normalization operations on the highest-scoring translated ID sequence to obtain the target text "Die Transformer-Architektur ist In this embodiment, the source language is English and the target language is German.

[0093] The above German text is written into a memory buffer as a UTF-8 byte stream and then transmitted via the bus to:

[0094] a) Mobile terminal display screen, or

[0095] b) The wireless module returns the data to the remote client in JSON format, or

[0096] c) The speech synthesis chip completes the TTS broadcast.

[0097] This completes a full end-to-end machine translation.

[0098] The method proposed in this embodiment also includes:

[0099] During the training phase, in each forward propagation, the activation value of each neuron (e.g., the output of activation functions such as ReLU and Sigmoid) is calculated. The top 50% of neurons are selected based on their activation values, and gradient updates are performed only on these activated neurons. This method not only reduces computation by only calculating the top 50% of neurons, thus reducing computational and memory usage, but also improves training efficiency and accelerates the training process, especially on large-scale datasets.

[0100] During the inference phase, in each forward propagation, the activation value of each neuron is calculated, and the top 30% of neurons in terms of activation value are selected. Only the selected neurons are used to calculate the sparse activation hidden state of the current layer, and then immediately sent to the next layer decoder to continue sequence generation until the translation is output.

[0101] This method not only significantly reduces computational load—computing only 30% of neurons during the inference phase, greatly reducing computational and memory consumption—but also improves inference speed: it significantly accelerates the model's inference process, making it suitable for real-time applications.

[0102] Furthermore, this strategy can be tailored to different tasks and data types. For example, when processing low-frequency words, neurons related to word meaning are activated. Low-frequency words often contain rich semantic information but occur infrequently. Selectively activating neurons related to these meanings allows for better capture of their semantic features. When processing high-frequency words, neurons related to grammatical structure are activated. High-frequency words often play a crucial role in grammatical structure, but their semantic information may be relatively simple. Selectively activating neurons related to grammatical structure allows for more efficient processing of these words. By activating the top 50% of neurons during training and retaining only 30% of highly activated neurons during inference, redundant computation can be significantly reduced, improving the training and inference efficiency of the model. This approach reduces computational resource consumption while maintaining model performance, making it particularly suitable for applications involving large-scale data and complex models.

[0103] Example 2

[0104] This invention proposes a semi-supervised machine translation system based on multi-temperature self-distillation and lightweight synergy, corresponding to the method in Example 1, comprising:

[0105] The training set creation module is used to collect text sequences to form a training set. It converts the text sequences in the training set into word sequences through a word embedding layer and adds positional encoding to the word sequences.

[0106] The knowledge distillation module is used to input the position-encoded lexical sequence into the teacher model; knowledge distillation is performed between the teacher model and the student model; the knowledge distillation process includes: performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels; both the teacher model and the student model extract lightweight intermediate representations through a grouped query attention mechanism; the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 are projected onto the same dimension after layer normalization, so that the teacher model and the student model are aligned; the student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss to complete the knowledge distillation.

[0107] The target language generation module receives the source sentence to be translated, encodes the source sentence into a sequence of sub-words using a word segmenter, and maps it to a sequence of vocabulary IDs; after padding the vocabulary ID sequence to a fixed length, it is fed into the student model that has completed knowledge distillation, and outputs the target language probability distribution; the target language probability distribution is decoded and textualized to obtain the target language text;

[0108] The physical output module is used to output the target language in text or speech form.

[0109] The implementation methods of each module and its function in the system are completely consistent with the steps of the method in Implementation Example 1, so they will not be repeated here.

[0110] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy, characterized in that, Includes the following steps: Step 1: Collect text sequences to form a training set, convert the text sequences in the training set into word sequences through a word embedding layer, and add positional encoding to the word sequences; Step 2: Input the position-encoded word sequence into the teacher model; Knowledge distillation between teacher and student models; The knowledge distillation process includes: performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels; both the teacher and student models extract lightweight intermediate representations through a grouped query attention mechanism; projecting the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 onto the same dimension after layer normalization, thus aligning the teacher and student models; and updating the student model's parameters through joint loss using the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals to complete the knowledge distillation. Step 3: Input the source sentence to be translated, use a word segmenter to encode the source sentence into a sequence of sub-words, and map it to a sequence of vocabulary IDs; Step 4: After filling the vocabulary ID sequence to a fixed length, feed it into the student model that has completed knowledge distillation, and output the target language probability distribution; Step 5: Decode and textify the probability distribution of the target language to obtain the target language text.

2. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, In step 1, adding positional encoding to the word sequence specifically involves: Position encoding is generated using sine and cosine functions: Where pos is the position index, i is the dimension index, and d is the position index. model For the embedding dimension, PE (pos,2i) and PE (pos,2i+1) These represent the sine and cosine position codes of position pos in dimensions 2i and 2i+1, respectively, which are used to provide the teacher model with the position information of words in the word sequence.

3. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, In step 2, the step of performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels specifically involves: In the first 2000 steps of training, the current teacher model parameters are saved every 100 steps as an instantaneous teacher, retaining only the output probability distribution and intermediate representations for the current training step. The probability distribution is calculated using the softmax function and used as a soft label, as shown in the following formula: In the formula, x represents the tensor composed of the word sequence encoded by the position of the input at the current training step, and P t (x) represents the teacher model's processing of input x at temperature T after training steps t. t The softened output probability distribution, i.e., the soft labels used for knowledge distillation; softmax(·) denotes the softmax function, z t (x) represents the teacher model's original output to input x at training step t, i.e., the vocabulary dimension real vector logits, where each dimension corresponds to the unnormalized prediction score of the corresponding word in the vocabulary; T t It is the temperature coefficient, when T t When T = 4, it is defined as high temperature. The probability distribution of the output at this time is defined as the high temperature soft label. t When T = 2.5, it is defined as medium temperature. The probability distribution of the output at this time is defined as a medium-temperature soft label. t When = 1, it is defined as low temperature, and the output probability distribution is defined as low temperature soft label.

4. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, In step 2, both the teacher model and the student model extract lightweight intermediate representations through a grouped query attention mechanism, specifically as follows: Both the teacher and student models execute a grouped query attention mechanism using the following formula to extract lightweight intermediate representations: H gqa =GQA(Q,K,V)=concat(head1,...,head j ,...,head n )W O In the formula, H gqa It is a lightweight intermediate representation. GQA stands for Grouped Query Attention mechanism, which divides the query header into n groups and shares the key K and value V in each group. Q is the query vector, which represents the query direction of the teacher model and the student model on the input data. K is the key vector, which represents the features of the input data. V is the value vector, which represents the actual content of the input data. head j W represents the attention weighted vector generated by the j-th query header, where concat represents the concatenation operation. O It is a projection matrix used to project the concatenated vectors onto the target dimension, generating the final lightweight intermediate representation H. gqa ; The formula for calculating the attention weighted vector of the j-th query header is as follows: In the formula, softmax(·) represents the softmax function, and Q j This is the j-th query header, where K and V refer to the shared key and value of the query headers, respectively. The superscript T indicates transpose, and d is the dimension of the key vector. It is a scaling factor used to prevent gradient vanishing.

5. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, In step 2, the step of projecting the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 onto the same dimension after layer normalization, so that the teacher model and the student model are aligned, specifically involves: First, lightweight intermediate representations of the l-th layer of the student model. With student model weights W s Multiplication, lightweighting the l+1th intermediate representation of the teacher model. Teacher model weights W t Multiply them, and then perform layer normalization separately to map them to the same feature space; Calculate the local distillation loss, which is only used to update the student model weights W. s and subsequent layer parameters of the student model, and weights W of the teacher model. t The formula for partial distillation loss while maintaining freezing is as follows: In the formula, L local This represents the local distillation loss, used to measure the difference between the student model and the teacher model at a certain layer output. D represents the dimension of the feature, i.e. the number of elements in each lightweight intermediate representation.

6. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, In step 2, the student model uses the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals, and updates its own parameters through joint loss as follows: The student model uses the high-temperature soft label, medium-temperature soft label, low-temperature soft label, and lightweight intermediate representation H generated by the teacher model. gqa As a monitoring signal, its own parameters are updated through a joint loss; the joint loss is the sum of the local distillation loss and the task cross-entropy loss.

7. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, The method further includes: During the training phase, in each forward propagation, the activation value of each neuron is calculated, and the top 50% of neurons in terms of activation value are selected. Gradient updates are performed only on these activated neurons. During the inference phase, in each forward propagation, the activation value of each neuron is calculated, and the top 30% of neurons in terms of activation value are selected. Only the selected neurons are used to calculate the sparse activation hidden state of the current layer, and then immediately sent to the next layer decoder to continue sequence generation until the translation is output.

8. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, Step 4 is as follows: After filling the vocabulary ID sequence to a fixed length, it is fed into the student model that has completed knowledge distillation, and forward computation is performed. The last layer outputs a logits vector sequence. The softmax function with a temperature coefficient of 1 is applied to the logits vector sequence to obtain the target language probability distribution.

9. The semi-supervised machine translation method based on multi-temperature self-distillation and lightweight synergy as described in claim 1, characterized in that, Step 5 specifically involves: Perform Beam Search decoding on the target language probability distribution, setting the beam width beam=4 and the length penalty lengthpenalty=1; Search for the candidate translation ID sequence with the highest score within the maximum length limit, and perform BPE merge character restoration, subtotalization, and punctuation normalization operations on the highest-scoring translation ID sequence to obtain the target language text.

10. A semi-supervised machine translation system based on multi-temperature self-distillation and lightweight synergy, characterized in that, include: The training set creation module is used to collect text sequences to form a training set. It converts the text sequences in the training set into word sequences through a word embedding layer and adds positional encoding to the word sequences. The knowledge distillation module is used to input the position-encoded lexical sequence into the teacher model; Knowledge distillation between teacher and student models; The knowledge distillation process includes: performing multi-temperature knowledge distillation in the teacher model to generate multi-temperature soft labels; both the teacher and student models extract lightweight intermediate representations through a grouped query attention mechanism; projecting the lightweight intermediate representations of the student model's layer l and the teacher model's layer l+1 onto the same dimension after layer normalization, thus aligning the teacher and student models; and updating the student model's parameters through joint loss using the multi-temperature soft labels and lightweight intermediate representations generated by the teacher model as supervision signals to complete the knowledge distillation. The target language generation module receives the source sentence to be translated, encodes the source sentence into a sequence of sub-words using a word segmenter, and maps it to a sequence of vocabulary IDs; after padding the vocabulary ID sequence to a fixed length, it is fed into the student model that has completed knowledge distillation, and outputs the target language probability distribution; the target language probability distribution is decoded and textualized to obtain the target language text; The physical output module is used to output the target language in text or speech form.

Images