Low-resource language translation method and system based on deconstruction distillation

Abstract

The invention relates to the technical field of language translation, in particular to a low-resource language translation method and system based on deconstruction distillation, and the method comprises the steps: obtaining parallel corpus data of a low-resource language and a general language; the method comprises the following steps: constructing a teacher model by taking parallel corpus data as input, constructing a student model trunk based on a pre-trained BERT big language model and optimizing the student model, calculating cross-task attention alignment loss based on the teacher model and the student model, and outputting and executing logits distillation based on the teacher model and the student model. Deployment of low-resource language translation is completed based on the trained student model, a high-quality knowledge migration source is provided for the student model by constructing the BERT teacher model subjected to full-parameter fine adjustment, and meanwhile, the problem of translation precision caused by insufficient low-resource language data is effectively solved by means of dual supervision of cross-task attention alignment and logits distillation.

Description

Technical Field

[0001] The present invention relates to the field of language translation technology, and in particular to a low-resource language translation method and system based on deconstruction distillation. Background Art

[0002] With the advancement of globalization, the demand for translation of low-resource languages ​​is increasing. They have important practical value in cross-border communication, cultural heritage, and education popularization. However, low-resource languages ​​are often limited by the scarcity of corpus and insufficient model training data, which leads to limited performance of existing translation systems. At the same time, due to the limitations of computing resources and storage capacity of edge devices such as portable translation terminals, large-scale pre-trained language models are difficult to deploy directly, and lightweight models are difficult to capture the complex semantic features of low-resource languages ​​due to parameter size limitations, resulting in low translation accuracy and poor generalization ability.

[0003] Existing efficient parameter fine-tuning technologies such as hybrid expert low-rank adaptation (MoLoRA) can reduce model parameters to a certain extent, but the lower projection matrix of the LoRA expert module is highly redundant, and the feature transformation paths between multiple tasks have a high degree of overlap, resulting in large memory usage and long inference time, which cannot meet the lightweight requirements of edge devices. Traditional knowledge distillation methods mostly rely on the logits transfer of the output layer, ignoring the semantic modeling strategy of the teacher model's intermediate layer attention mechanism. In low-resource language translation, there are significant differences in the grammatical structure and semantic association of different languages. It is difficult for the student model to capture the deep alignment logic of the teacher model through a single output layer distillation, resulting in semantic breaks in the translation process. Low-resource language translation often involves multiple scenarios. Existing methods lack a mechanism for dynamically adjusting the distillation weights and cannot allocate resources according to the translation difficulty of different scenarios. At this stage, a low-resource language translation method and system based on deconstruction distillation is needed. Summary of the Invention

[0004] In order to solve the problems of inaccurate resource allocation for different scenarios and poor adaptability to edge devices in existing methods, the present invention provides a low-resource language translation method and system based on deconstructive distillation.

[0005] In a first aspect, the present invention provides a low-resource language translation method based on deconstructive distillation, which adopts the following technical solutions: A low-resource language translation method based on deconstructive distillation, comprising: Obtain parallel corpus data for low-resource languages ​​and common languages; The teacher model is constructed using parallel corpus data as input, including using the BERT large language model as the architecture and pre-training the BERT large language model using full parameter fine-tuning to obtain the teacher model; The student model backbone is built based on the pre-trained BERT large language model and optimized. This includes lightweighting the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method. Calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices through the L2 norm; Perform logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss; Deploy low-resource language translation based on the trained student model.

[0006] Furthermore, the teacher model is constructed using parallel corpus data as input, including the BERT large language model as the basic architecture. The BERT large language model includes an L-layer Transformer encoder, each layer of which is composed of a multi-head self-attention mechanism and a feedforward neural network. The low-resource language text is used as the input sequence and the corresponding general language text is used as the target output sequence. The input sequence is processed into a token sequence by a word segmenter and mark symbols are added, and then converted into a fixed-length word embedding vector sequence.

[0007] Furthermore, the full parameter fine-tuning is used to pre-train the BERT large language model, including encoding the word embedding vector sequence through the BERT large language model to obtain a context representation, mapping the context representation to the vocabulary probability distribution of the target language through the output layer, defining the cross entropy loss function according to the vocabulary probability distribution, and minimizing the cross entropy loss function through the gradient descent algorithm. The vocabulary probability distribution expression is: , in, Indicates that the teacher model parameters are Under the condition of , given the input sequence x, the conditional probability distribution of the output target sequence y is, represents the normalization function, represents the weight matrix of the output layer, It is represented as the context representation obtained after the input sequence x is encoded by the BERT model. Represented as the bias vector of the output layer.

[0008] Furthermore, the student model is optimized using a structural deconstruction low-rank perturbation method, including compressing the input low-resource language text features through a shared encoding matrix to generate universal features, then defining a task-conditional decoding path, obtaining a task embedding vector through language ID mapping, and using a structural control network to map the task embedding vector into an intermediate matrix, and multiplying it with a shared decoding kernel to generate a structural control decoding matrix, and generating an overall perturbation based on the structural control decoding matrix and the universal features.

[0009] Furthermore, the attention matrices of the corresponding layers of the teacher model and the student model are extracted, including all translation tasks based on the parallel corpus, selecting the middle layers of the teacher model and the student model and all the attention heads of each layer, and extracting the attention matrices of the teacher model and the student model at the first The attention matrix of the layer, the elements of the attention matrix represent the strength of the association between two basic units in the input text.

[0010] Furthermore, the cross-task attention alignment loss is calculated based on the teacher model and the student model, including calculating the L2 norm distance between the attention matrices of the teacher model and the student model as the local structure alignment loss. A head-by-head alignment strategy is adopted to select all attention heads for the multi-head attention mechanism of each layer, perform alignment calculation on each attention head separately, and take the average of the losses of all heads. The alignment loss expression is: , Where n represents the total number of low-resource language translation tasks, L represents the number of model intermediate layers involved in alignment, and H represents the number of attention heads in each layer. represents the teacher model in the i-th low-resource language translation task In, The attention matrix of the h-th attention head in the layer, Indicates that the student model is in the i-th low-resource language translation task, The attention matrix of the h-th attention head in the layer.

[0011] Furthermore, the logits distillation is performed based on the outputs of the teacher model and the student model, including obtaining the original outputs of both the teacher model and the student model, generating a softened probability distribution based on the scaling mechanism of the original outputs using a temperature parameter, and quantifying the difference in the softened probability distributions between the teacher and student models by KL divergence to obtain the logits distillation loss, which is expressed as: , in, and They represent the predicted probabilities of the teacher model and the student model for the cth word, respectively, and C represents the number of vocabulary categories.

[0012] Furthermore, the logits distillation is performed based on the outputs of the teacher model and the student model, and the task weights are calculated based on the task loss and the task average loss, and the task weights are normalized. Based on the principle of multi-loss collaborative supervision, the distillation loss is integrated with the alignment loss and the loss of each language task through the total loss function. The loss of each language task is the cross entropy loss of the student model on the low-resource language translation task. The total loss function formula is: , in, represents a hyperparameter used to adjust the weight of distillation loss in the total loss. represents the logits distillation loss, represents the cross-task attention alignment loss, represents the hyperparameter used to adjust the weight of the cross-task attention alignment loss, represents the dynamic weight of the i-th low-resource language translation task.

[0013] Furthermore, the deployment of low-resource language translation is completed based on the trained student model, including pre-deployment optimization of the trained student model, converting the model parameters from 32-bit floating point numbers to 8-bit integers using the INT8 quantization method, and implementing the quantization process through linear mapping to convert the model into a format for end-side deployment, and completing the deployment by eliminating redundant computing nodes and fusing operators.

[0014] The second aspect is a low-resource language translation system based on deconstructive distillation, including: The data acquisition module is configured to: acquire parallel corpus data of low-resource languages ​​and common languages; The teacher model module is configured to: construct a teacher model using the parallel corpus data as input, including pre-training the BERT large language model using the BERT large language model as the architecture and fine-tuning the BERT large language model using all parameters to obtain the teacher model; The student model module is configured to build the student model backbone based on the pre-trained BERT large language model and optimize the student model, including lightweight processing of the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method; The alignment module is configured to calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices using the L2 norm; The optimization module is configured to perform logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss; The transformation module is configured to: complete the deployment of low-resource language translation based on the trained student model.

[0015] In summary, the present invention has the following beneficial technical effects: 1. This paper constructs a BERT teacher model that has been fine-tuned in all parameters and utilizes its deep learning ability for parallel corpora to provide a high-quality knowledge transfer source for the student model. At the same time, with the dual supervision of cross-task attention alignment and logits distillation, the lightweight student model can not only replicate the intermediate-layer semantic parsing strategy of the teacher model, but also accurately match the output-layer vocabulary prediction distribution, effectively compensating for the translation accuracy problem caused by insufficient low-resource language data, especially in language translation tasks with complex grammar and scarce data.

[0016] 2. The present invention introduces dynamic task weights and adaptively adjusts the supervision intensity according to the loss of each language task, so that the model prioritizes high-difficulty tasks, improves the global adaptability to different low-resource languages, shares the encoding matrix and decoding core to promote cross-language knowledge transfer, avoids information isolation between tasks, and eliminates the need for the model to be trained from scratch when a new language task is added. Instead, the model only needs to update the task embedding vector for rapid adaptation.

[0017] 3. The student model of the present invention reduces the number of parameters by reducing the number of layers and lowering the hidden dimensions. Combined with INT8 quantization and operator optimization, it significantly reduces memory usage and inference latency, and can be efficiently deployed on edge devices. The structural deconstruction low-rank perturbation method further reduces parameter redundancy. In multi-task scenarios, the parameter growth rate is much lower than that of traditional methods, taking into account both model efficiency and cross-language adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the overall process of a low-resource language translation method based on deconstruction distillation according to an embodiment of the present invention.

[0019] Figure 2 This is a framework diagram of a low-resource language translation method based on deconstruction distillation in an embodiment of the present invention.

[0020] Figure 3 It is a framework diagram of the structural deconstruction low-rank perturbation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] Example 1 Reference Figure 1 , a low-resource language translation method based on deconstruction distillation in this embodiment includes: Obtain parallel corpus data for low-resource languages ​​and common languages; The teacher model is constructed using parallel corpus data as input, including using the BERT large language model as the architecture and pre-training the BERT large language model using full parameter fine-tuning to obtain the teacher model; The student model backbone is built based on the pre-trained BERT large language model and optimized. This includes lightweighting the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method. Calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices through the L2 norm; Perform logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss; Deploy low-resource language translation based on the trained student model.

[0023] Specifically, a low-resource language translation method based on deconstructive distillation includes the following: S1. Obtain parallel corpus data for low-resource languages ​​and common languages; like Figure 1 、 Figure 2 As shown in the figure, the scope of the target low-resource languages ​​is clarified. The low-resource languages ​​refer to languages ​​with less available annotated data in the field of natural language processing. Common languages ​​are widely used and resource-rich languages, such as Chinese and English. Secondly, parallel corpus data of the low-resource languages ​​and common languages ​​are collected through various channels. These channels include but are not limited to public language resource libraries, bilingual comparison texts accumulated by professional translation agencies, and bilingual data collected by crowdsourcing platforms that have been manually verified. Ensure that the collected parallel corpus covers multiple fields such as daily conversations, news reports, academic literature, and life services to improve the diversity and coverage of the corpus.

[0024] Next, the collected original parallel corpus is preprocessed. The preprocessing process includes removing noise data in the text, unifying the text format, and performing sentence-level alignment. That is, by comparing the length and calculating the vocabulary overlap, the sentences in the low-resource language are matched one by one with their corresponding common language translation sentences to form sentence-level parallel language pairs. For existing long sentences, they are split into semantically complete short sentences to improve the alignment accuracy, while the short sentences are kept as they are.

[0025] Finally, the processed parallel corpus is divided into training set, validation set and test set according to the preset ratio of 7:2:1. The training set is used for the subsequent training process of the teacher model and student model, the validation set is used to monitor the performance changes during the model training process to adjust the hyperparameters, and the test set is used to finally evaluate the translation effect of the model. All corpora are stored in a structured format to facilitate calling and processing in subsequent steps.

[0026] S2. Build a teacher model using the parallel corpus data as input, including using the BERT large language model as the architecture and pre-training the BERT large language model using full parameter fine-tuning to obtain the teacher model; A teacher model is constructed using the parallel corpus data of low-resource languages ​​and general languages ​​obtained in step S1 as input. The teacher model is based on the BERT large language model. The BERT large language model includes L layers of Transformer encoders. In this embodiment, L=12 layers, which can be selected according to the complexity of the low-resource language. Each layer of Transformer encoder consists of a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism includes H attention heads, which are used to capture the correlation between tokens in the input sequence from different semantic dimensions. The feedforward neural network consists of two linear transformation layers and a GELU activation function in the middle, which is used to perform feature conversion and enhancement on the output of the attention mechanism.

[0027] During model construction, low-resource language text is used as the input sequence, and the corresponding general language text is used as the target output sequence. The input sequence is first processed using a low-resource language tokenizer adapted for the BERT model to break the input sequence into tokens, the smallest semantic units. This forms a token sequence. A special symbol [CLS] is added to the beginning of the token sequence to indicate the overall semantics of the sequence, and a special symbol [SEP] is added to the end to mark the end of the sequence. Padding is then used to adjust the token sequence to a fixed length, for example, 512 tokens. Sequences that are shorter than this length are padded with "[PAD]," and sequences that are longer are truncated to this length.

[0028] The processed token sequence is then converted into a word embedding vector sequence. Specifically, each token is mapped to a corresponding word vector through a word embedding table. At the same time, position embedding is incorporated to represent the position information of the token in the sequence and paragraph embedding is used to distinguish sequences in different languages. The three are superimposed to form the final word embedding vector sequence. The dimension of this vector sequence is [fixed length, hidden layer dimension].

[0029] The BERT large language model is then pre-trained using full parameter fine-tuning. The word embedding vector sequence is input into the L-layer Transformer encoder of the BERT large language model. After layer-by-layer encoding, the contextual representation H of the input sequence is obtained. The processing process of each layer of the Transformer encoder is as follows: the word embedding vector sequence is first calculated by the multi-head self-attention mechanism to obtain the attention-weighted features. The calculation of the multi-head self-attention mechanism is based on the formula: , Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively, which are obtained by linear transformation of input features. For the dimension of each attention head, softmax is a normalization function used to convert the similarity between tokens into attention weights. Subsequently, the feature is residually connected with the input feature and normalized by the layer to obtain the output of the multi-head self-attention mechanism, which is then input into the feedforward neural network. After the first linear transformation layer dimension is mapped from the hidden layer dimension to 4 times the hidden layer dimension, the GELU activation function and the second linear transformation layer dimension are mapped back to the hidden layer dimension from 4 times the hidden layer dimension, the residual connection is performed with the input of the feedforward neural network and normalized by the layer to obtain the output of the current Transformer layer as the input of the next layer. The context representation H output by the encoder is processed by the output layer and mapped to the vocabulary probability distribution of the target language. The calculation process of the output layer is based on the formula: , in, Indicates that the teacher model parameters are Under the condition of , given the input sequence x, the conditional probability distribution of the output target sequence y is, represents the normalization function, represents the weight matrix of the output layer, It is represented as the context representation obtained after the input sequence x is encoded by the BERT model. Represented as the bias vector of the output layer.

[0030] During the pre-training process, the cross entropy loss function is defined based on the vocabulary probability distribution and the true label of the target output sequence, that is, the token sequence corresponding to the general language text. The formula is: , in, is the one-hot vector of the true label, the correct token corresponding position is 1, and the rest are 0, It is expressed as a set of teacher model parameters. The formula is used to quantify the difference between the model prediction probability and the true label. Then, the gradient descent algorithm is used to minimize the cross entropy loss function and iteratively update all parameters of the teacher model. In each iteration, a batch of data from the training set is used to update the parameters, and the BLEU score of the model is monitored on the validation set. When the BLEU score on the validation set does not improve for multiple epochs, in this embodiment, 3 epochs are used, the training is stopped, and the trained teacher model is finally obtained.

[0031] S3. Build the student model backbone based on the pre-trained BERT large language model and optimize the student model, including lightweight processing of the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method; like Figure 3 As shown, based on the BERT large language model pre-trained in step S2, the student model backbone is constructed and the student model is optimized. First, the BERT large language model is lightweighted, and the BERT model of the same series as the teacher model is selected as the student model backbone. By reducing the number of layers of the Transformer encoder from 12 to 6, reducing the hidden layer dimension from 768 to 512 dimensions, and reducing the number of attention heads from 12 to 6, while ensuring that the model retains core semantic encoding capabilities, such as understanding the basic vocabulary and grammatical structure of low-resource languages, the overall parameter amount is reduced by about 60% to adapt to resource-constrained scenarios such as edge devices, forming the basic backbone structure of the student model.

[0032] Subsequently, a structural deconstruction-based low-rank perturbation method is used to optimize the student model to enhance its adaptability to multiple low-resource language tasks and reduce parameter redundancy. The specific process is as follows: the input feature dimension is defined as n, which is the feature dimension of the low-resource language text output by the student model backbone; the output feature dimension is defined as m, which is the feature dimension of the universal language; the perturbation rank is defined as r, which is used to control the low-rank space dimension; the low-resource language task embedding dimension is k, and the intermediate dimension is h. The overall perturbation is expressed as the core formula: , in, is the input feature, The encoding matrix shared by all tasks is used to compress input features. To activate the transformation function, it is used to improve the expressiveness and stability of the projection space. Indicates a task The structure controls the decoding matrix, which is implemented through three links: shared compression path, task condition decoding path and disturbance integration: In the shared compression path, the shared encoding matrix Shared by all low-resource language tasks, used to extract common language features from input features and low-resource language text features output by the student model backbone Compress, Generate universal features after activation function processing , corresponding to the formula Part, to achieve shared compression and nonlinear transformation of input features; in the task condition decoding path, for the i-th low-resource language translation task, through language ID mapping, such as assigning ID = 0 to Tibetan and assigning ID = 1 to Swahili, a unique task embedding vector is obtained , which encodes language-specific information, such as grammatical rules and vocabulary collocation habits, using the structural control network Embedding the task into a vector Mapped to an intermediate matrix, the structure control network is implemented using a two-layer fully connected network. The first layer maps the task embedding vector from dimension k to the intermediate dimension h. The formula is: , in, is the first layer weight matrix, is the first layer bias vector, is the activation function, and the second layer maps the intermediate result from dimension h to dimension r. The formula is ,in, is the second layer weight matrix, is the second layer bias vector, and finally the intermediate matrix is ​​obtained , and then share it with the shared decoding core Multiplied together, the shared decoding kernel represents the basic decoding matrix shared by all tasks, unifies the semantic mapping rules, and generates a structural control decoding matrix: , in, is the task embedding vector, obtained by mapping the task ID, For the structural regulatory network, the task embedding is mapped into a structural control matrix , is the decoding weight kernel shared by all tasks, that is, , which incorporates the task-specific information of a specific low-resource language; in the perturbation integration step, the overall perturbation is generated based on the structural control decoding matrix and the universal features. The overall perturbation is: ,pass and The matrix multiplication of is obtained, which is consistent with the core formula Completely corresponding, it realizes the superposition of task-specific perturbations on the basis of general features, which is different from the traditional LoRA module, that is, each task requires the introduction of parameter quantities. The total number of parameters is , Compared with the number of tasks, the total parameter calculation formula of this method is: , in, Structural regulatory network The number of parameters, Corresponding shared coding matrix The number of parameters, where r is the perturbation rank, controls the low-rank space dimension, n is the input feature dimension, m is the output feature dimension, and h is the intermediate feature dimension. is the total number of low-resource language tasks, Embedding vector for each task Dimension, when the number of tasks N is large and When , the parameter scale is significantly reduced, effectively reducing parameter redundancy; at the same time, the design provides a continuous differentiable function mapping from the task embedding space to the perturbation matrix , which can be optimized collaboratively with the backbone distillation objective during training; the final output of the student model is the superposition of its backbone output and the overall perturbation, i.e. ,in, The basic results of the student model backbone output enable the model to dynamically optimize the output for different low-resource languages. Finally, the lightweight backbone is combined with the structural deconstruction low-rank perturbation module to form a complete student model. During initialization, the backbone parameters are inherited from the corresponding layer parameters of the pre-trained BERT, and the perturbation module parameters include and all task embedding vectors Random initialization lays the foundation for subsequent fine-tuning through attention alignment and knowledge distillation.

[0033] S4. Calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices through the L2 norm;

[0034] Based on the teacher model trained in step S2 and the student model constructed in step S3, the cross-task attention alignment loss is calculated to achieve effective migration of semantic information at the intermediate structural level. The specific process is as follows: First, for all low-resource language translation tasks covered by the parallel corpus in step S1, the intermediate layers of the model involved in attention alignment and the attention heads of each layer are determined, and the intermediate layers that can reflect the core semantic modeling strategy in the teacher model and the student model are selected. In this embodiment, the 3rd to 8th layers of the Transformer encoder are selected, with a total of L=6 layers, and all the attention heads of each layer are included, with H=12 heads per layer, to ensure that the alignment supervision covers the key semantic processing stage of the model; then, for each low-resource language translation task , respectively extract the attention matrices of the teacher model and the student model at the hth attention head in the lth layer, where the attention matrix of the teacher model is recorded as , the attention matrix of the student model is recorded as , the element values of these attention matrices represent the association strength between two tokens (basic semantic units) in the input text. The larger the value, the closer the association. For example, in the Tibetan-Chinese translation task, the matrix elements may reflect the grammatical dependency between " " (I) and "my"; Next, the local structure alignment loss is calculated using the head-by-head alignment strategy. The alignment calculation is performed separately for each attention head. The difference between the teacher model and the student model's corresponding attention matrices is measured by the L2 norm. The calculation formula is , where represents the square of the L2 norm, which is used to quantify the overall difference between the two matrices at the element level; After that, the local losses of all tasks, all intermediate layers participating in the alignment, and all attention heads are aggregated to obtain the cross-task attention alignment loss. Its expression is: , , where n represents the total number of low-resource language translation tasks, L represents the number of intermediate layers of the model participating in the alignment, H represents the number of attention heads in each layer, represents the attention matrix of the teacher model in the i-th low-resource language translation task in the layer and the h-th attention head, represents the attention matrix of the student model in the i-th low-resource language translation task, in the [[ID=XX]] layer and the h-th attention head. By normalizing the result of the triple summation by nL, the contributions of different tasks, layers, and attention heads to the total loss are balanced; The design of this loss function is based on the characteristics of the attention matrix in the Transformer architecture. It not only explicitly models the dependencies between tokens but also implicitly reflects the semantic modeling preferences. By forcing the attention distribution of the student model to align with that of the teacher model, it avoids the problems of shallow alignment and deep offset caused by distillation only through the output layer, ensuring the effective transfer of the semantic modeling strategy in the intermediate layer; Finally, the calculated cross-task attention alignment loss is used for the fusion of the total loss function in the subsequent step S5, and together with the distillation loss and the task loss, it co-supervises the training of the student model, enabling the student model to inherit the knowledge of the teacher model's output layer while simultaneously learning the structure-level semantic processing ability of the intermediate layer.

[0035] S5. Perform logits distillation based on the outputs of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distributions of the teacher model and the student model, and calculating the KL divergence as the distillation loss; It should be noted that in the translation, the specific content of , , etc. are left unchanged as they are likely specific identifiers or notations in the original text. If there is more context available for these tags, a more accurate translation might be possible.​After the teacher model and student model are trained, logits distillation is performed and multi-loss supervision is integrated to optimize the student model. First, after completing the cross-task attention alignment in step S4, the representation of the final output of the teacher model is transferred to the student model through logits distillation, forming a dual knowledge transfer path of structure transfer and output alignment, where the teacher model is denoted as T and the student model is denoted as S. In logits distillation, the original outputs of the teacher model and the student model are first obtained, that is, the original logits output of the teacher model and the raw logits output of the student model , the original output is scaled based on the temperature parameter to generate a softened probability distribution. The temperature parameter τ is set to 2.0. A high temperature τ helps the teacher model T output a smoother probability distribution, thereby alleviating the gradient explosion problem in the learning process of the student model S. The softened probability distribution calculation formula of the teacher model is , the calculation formula of the softened probability distribution of the student model is ,in, is the normalization function that converts the scaled logits into a probability distribution, Model for teachers The raw logits output of Model for students The original logits output, τ is the temperature parameter, and then the difference in the softened probability distribution between the teacher and student models is quantified by KL divergence to obtain the logits distillation loss, which is expressed as: , in, and Denote the predicted probabilities of the teacher model and the student model for the cth word, respectively. C represents the number of word categories. The loss function is used to measure the difference between the two probability distributions and promote the student model to imitate the output distribution of the teacher model. In order to further improve the task adaptability of the student model, the task adaptive distillation weight is introduced. Specifically, suppose that in the current tth training round, the task The loss is , the average loss of all tasks is , then the task weight update rule is defined as: , in, is a hyperparameter that controls the rate at which weights are adjusted, Represents the average loss of all tasks. When the loss of a task When it is higher than the average, it indicates that the task is more difficult or the model performs poorly on this task. Will increase, otherwise it will decrease. In addition, the weight update mechanism conforms to the exponential decay paradigm of change stability. When the task loss tends to the average loss, ,in, Represents the limit condition, that is, when the loss of the i-th task Infinitely close to the average loss of all tasks hour, represents the updated weight of the i-th task during the t+1-th round of training, represents the current weight of the i-th task in the t-th round of training, which means that the task weight tends to be stable after convergence, avoiding the drastic fluctuation of the weight in the late stage of training, which is conducive to the final global optimization. In order to ensure that the sum of all task weights is 1 and prevent overfitting of local tasks, all task weights need to be normalized. The formula is: , in, is the sum of the weights of all tasks after round t of training, and the normalized weights are ensured by dividing the weight of each task by the sum of the weights of all tasks. satisfy .

[0036] Finally, based on the principle of multi-loss collaborative supervision, the distillation loss is integrated with the alignment loss and the loss of each language task through the total loss function. The loss of each language task is the cross entropy loss of the student model on the low-resource language translation task. , the total loss function formula is: , in, represents a hyperparameter used to adjust the weight of distillation loss in the total loss. represents the logits distillation loss, represents the cross-task attention alignment loss, represents the hyperparameter used to adjust the weight of the cross-task attention alignment loss, Represents the dynamic weight of the i-th low-resource language translation task, and the student model is trained by minimizing the total loss function to enable it to achieve the best performance in the target task. It achieves better performance on multi-tasks by co-optimizing structure-level alignment and output-level distillation, and improving the generalization ability of the model across multiple tasks.

[0037] S6. Complete the deployment of low-resource language translation based on the trained student model.

[0038] The trained student model is optimized by structural deconstruction low-rank perturbation and integrated with a lightweight model of multi-loss supervision. It is deployed for the actual application scenario of low-resource language translation. The specific process is as follows: First, the trained student model is optimized before deployment to adapt to the resource limitations of edge devices and local servers. The optimization process includes quantization processing and format conversion. The quantization process uses INT8 quantization technology to convert the model parameters from 32-bit floating point numbers to 8-bit integers, which can greatly reduce memory usage while controlling the accuracy loss. The quantization process is implemented through linear mapping. Let the original parameters be , quantized parameters ,in, is a rounding function, and 255 is the maximum value of an 8-bit integer. This formula linearly maps the original parameters to the 8-bit integer range, preserving the relative distribution characteristics of the parameters. In the format conversion phase, the model is converted into a format suitable for end-side deployment, and static graph analysis tools are used to eliminate redundant computing nodes and fusion operators. Specifically, this includes removing nodes used only for training, such as the dropout layer, the update nodes of the BatchNorm layer, and invalid operation nodes. At the same time, operators for continuous calculations are fused into a single composite operator to reduce the number of memory reads and writes during the calculation process and improve inference efficiency. Then, the corresponding inference engine is selected according to the hardware architecture of the deployment device. For mobile devices with ARM architecture, the TensorFlowLite or ONNXRuntimeMobile engine is used, taking advantage of its computing kernel optimized for low-power hardware.

[0039] For local servers with x86 architecture, ONNXRuntime or PyTorchJIT engines are used to support multi-threaded parallel reasoning and meet the low-latency requirements of hundreds of learners using it simultaneously. An end-to-end low-resource language translation process is then designed. In the input preprocessing stage, the low-resource language text input by the user is received and converted into a token sequence through the word segmenter consistent with the training stage. After adding special symbols "[CLS]" and "[SEP]", it is converted into a fixed-length word embedding vector. If the length is insufficient, zero is added, and if it is too long, it is truncated. In the model inference stage, the word embedding vector is input into the deployed student model, and is calculated by the lightweight backbone and structural deconstruction low-rank perturbation module to output the logits of the general language, which is converted into a vocabulary probability distribution through the softmax function. In the translation generation stage, a greedy decoding strategy (selecting the word with the highest probability as the current output) is used to generate the general language translation. For long texts, a sliding The dynamic window mechanism is used for block processing to ensure that the input length of each block does not exceed the maximum sequence length of the model. In this embodiment, it is set to 512 tokens. Finally, offline deployment and dynamic update are completed, and the optimized model, inference engine, low-resource language basic vocabulary and word segmentation rules are packaged to the edge device to support real-time translation in a network-free environment. When new parallel corpus is collected (such as the revised translation feedback from users), the lightweight fine-tuning of steps S3-S5 is performed, that is, only the parameters of the structural deconstruction low-rank perturbation module are updated to generate an incremental model, which is pushed to the device via OTA to avoid the high bandwidth consumption of the full model update; after deployment, the translation accuracy, inference delay (single sentence translation time) and memory usage of the model are monitored in real time, and the corresponding corpus fine-tuning task embedding vector is supplemented for abnormal situations, or the number of model layers is further compressed to balance efficiency and performance, ultimately realizing the efficient application of low-resource language translation models in scenarios such as educational equity and cross-language communication.

[0040] Example 2 The difference between this embodiment and embodiment 1 is that this embodiment provides a specific experiment of a low-resource language translation method based on deconstructive distillation; To demonstrate the feasibility of a low-resource language translation method based on deconstructive distillation, this example uses the representative dataset GLUE. The detailed information of this dataset is as follows: GLUE (General Language Understanding Evaluation) is a comprehensive natural language understanding (NLU) benchmark designed to evaluate the performance of language models across a variety of language understanding tasks. The benchmark dataset covers a range of challenging tasks, including CoLA (syntactic acceptability), SST-2 (sentiment analysis), MRPC (synonymous sentence identification), STS-B (semantic similarity assessment), QQP (duplicate question detection), MNLI (multi-domain natural language inference), QNLI (question natural language inference), RTE (text entailment identification), and WNLI (Winograd grammar challenge). These tasks are designed to comprehensively test the reasoning, generalization, and transfer capabilities of language models (LLMs) in language understanding. By aggregating weighted scores across tasks, the GLUE benchmark provides a unified evaluation framework, making it an important tool for comparing the effectiveness of different models, algorithms, and methods. Detailed statistical information about the dataset is shown in Table 1.

[0041] Table 1 GLUE dataset information table

[0042] Table 1 shows the GLUE dataset information. In the experiment of this embodiment, LLM: BERT and DeBERTaV3 are selected on the GLUE dataset using the FFT method, LoRA method, MoLoRA method, TinyBERT method, DistilBERT method, BERT-PKD method, BERT TINY Eight fine-tuning strategies are evaluated with the low-resource language translation method implemented in this paper, using DistilBERT-base as the student model of BERT-base for pre-training and fine-tuning.

[0043] This experiment not only compares the fine-tuning framework with FFT and LoRA, but also with methods that use knowledge distillation for model compression. This article will further explore the performance of various fine-tuning methods on different tasks through detailed experimental data analysis.

[0044] (1) FFT: This is a fine-tuning method that optimizes all parameters of a pre-trained model on a specific task to adapt it to the requirements of a new task.

[0045] (2) LoRA: This is a method for efficiently fine-tuning large-scale pre-trained models through low-rank matrix factorization, significantly reducing the number of trainable parameters.

[0046] (3) MoLoRA: This is an advanced method that extends LoRA and further improves the flexibility and efficiency of fine-tuning by integrating the modular design of MoE. MoLoRA-10 and MoLoRA-15 correspond to 10 and 15 experts respectively.

[0047] (4) TinyBERT: This is a fine-tuning method that uses a multi-stage training strategy based on knowledge distillation to optimize deep semantic representations through distillation learning to compress the BERT model.

[0048] (5) DistilBERT: This is a fine-tuning method that selectively reduces the number of model layers through shallow distillation and maintains deep semantic understanding capabilities through training.

[0049] (6) BERT-PKD: This is a fine-tuning method that transfers path information between layers of the teacher model through path knowledge distillation, reducing the model size while retaining the semantic representation performance.

[0050] (7) BERT TINY : This is a fine-tuning method that extremely compresses the BERT model by reducing the number of model layers and parameters, significantly reducing computing resource requirements while retaining its performance as much as possible.

[0051] (8) MobileBERT: This is a fine-tuning method that compresses the BERT model by combining knowledge distillation technology with low-rank factorization and lightweight deep bottleneck layer design.

[0052] Table 2 Comparative experimental table of the performance of different fine-tuning methods on the GLUE dataset

[0053] Table 3 Comparison of memory usage and inference time on the GLUE dataset

[0054] Tables 2 and 3 intuitively show the results of the comparative experiments. Table 2 is a comparative experiment on the performance of different fine-tuning methods on the GLUE dataset, and Table 3 is a comparison of memory usage and inference time on the GLUE dataset. The student model proposed in this embodiment, namely KD-MoLoRA, scored slightly lower than LoRA, MoLoRA, and FFT on the GLUE dataset, but higher than other traditional KD methods.

[0055] In comparative experiments, KD-MoLoRA in this embodiment achieved approximately 96.7% of the performance of MoLoRA-10 and approximately 95.4% of the performance of MoLoRA-15, improving model compactness by 15.8% and 20.9%, respectively, while reducing inference time by 7.3% and 6.0%, respectively. This result highlights the advantages of knowledge distillation. Specifically, a reasonable knowledge distillation strategy can compress the model so that the student model achieves performance similar to that of the teacher model, thereby effectively saving computing resources.

[0056] Compared with other fine-tuning methods using KD, the student model KD-MoLoRA in this embodiment shows a certain performance improvement. Specifically, on the GLUE multi-task dataset, the performance of KD-MoLoRA is better than that of BERT. TINY It has increased by 15.7%, and 11.7% compared to DistilBERT. This is mainly due to the fact that KD-MoLoRA integrates the MoE design, which can dynamically allocate appropriate experts using the expert routing mechanism, so that the model can flexibly cope with diverse tasks and achieve better performance. In addition, KD-MoLoRA can still maintain robust performance in resource-constrained scenarios. This is mainly due to its structural deconstruction low-rank perturbation design, which can reduce the redundancy of similar experts when extracting common features, ensuring the training quality of the model even in resource-constrained environments.

[0057] Compared with FFT, which updates all parameters, KD-MoLoRA also performs well. Specifically, on the GLUE dataset, KD-MoLoRA achieved about 95.7% of the performance of FFT, while the inference time was shortened by 14.6% and the structural compactness was improved by 76.1%. At the same time, KD-MoLoRA also achieved about 97.3% of the performance of LoRA, and the memory usage was reduced by 144.9MB, and the inference time was shortened by 13.5%. This series of improvements is because KD-MoLoRA not only uses KD for simple model compression, but also because the core of the KD-MoLoRA design in this embodiment is to capture the deep semantic information of the teacher model and identify the importance differences of different tasks through the cross-task attention alignment mechanism. Through the task-adaptive distillation optimization method, KD-MoLoRA can assign different weights according to the importance of the task, thereby achieving better performance in multi-task learning.

[0058] Example 3 The difference between this embodiment and embodiment 1 is that this embodiment provides a low-resource language translation system based on deconstructive distillation; The data acquisition module is configured to: acquire parallel corpus data of low-resource languages ​​and common languages; The teacher model module is configured to: construct a teacher model using the parallel corpus data as input, including pre-training the BERT large language model using the BERT large language model as the architecture and fine-tuning the BERT large language model using all parameters to obtain the teacher model; The student model module is configured to build the student model backbone based on the pre-trained BERT large language model and optimize the student model, including lightweight processing of the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method; The alignment module is configured to calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices using the L2 norm; The optimization module is configured to perform logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss; The transformation module is configured to: complete the deployment of low-resource language translation based on the trained student model.

[0059] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A low-resource language translation method based on deconstructive distillation, characterized in that: include: Obtain parallel corpus data for low-resource languages ​​and common languages; The teacher model is constructed using parallel corpus data as input, including using the BERT large language model as the architecture and pre-training the BERT large language model using full parameter fine-tuning to obtain the teacher model; The student model backbone is built based on the pre-trained BERT large language model and optimized. This includes lightweighting the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method. Calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices through the L2 norm; Perform logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss; Deploy low-resource language translation based on the trained student model.

2. A low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The teacher model is constructed using parallel corpus data as input, including the BERT large language model as the basic architecture. The BERT large language model includes an L-layer Transformer encoder, each layer of which is composed of a multi-head self-attention mechanism and a feedforward neural network. The low-resource language text is used as the input sequence and the corresponding general language text is used as the target output sequence. The input sequence is processed into a token sequence by a word segmenter and a mark symbol is added, and then converted into a fixed-length word embedding vector sequence.

3. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The full parameter fine-tuning is used to pre-train the BERT large language model, including encoding the word embedding vector sequence through the BERT large language model to obtain a context representation, mapping the context representation to the vocabulary probability distribution of the target language through the output layer, defining the cross-entropy loss function based on the vocabulary probability distribution, and minimizing the cross-entropy loss function through the gradient descent algorithm. The vocabulary probability distribution expression is: ， in, Indicates that the teacher model parameters are Under the condition of , given the input sequence x, the conditional probability distribution of the output target sequence y is, represents the normalization function, represents the weight matrix of the output layer, It is represented as the context representation obtained after the input sequence x is encoded by the BERT model. Represented as the bias vector of the output layer.

4. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The student model is optimized using a structural deconstruction low-rank perturbation method, which includes compressing the input low-resource language text features through a shared encoding matrix to generate universal features, then defining a task-conditional decoding path, obtaining a task embedding vector through language ID mapping, mapping the task embedding vector into an intermediate matrix using a structural control network, and multiplying it with a shared decoding kernel to generate a structural control decoding matrix, and generating an overall perturbation based on the structural control decoding matrix and the universal features.

5. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The attention matrices of the corresponding layers of the teacher model and the student model are extracted, including all translation tasks based on parallel corpus, selecting the middle layers of the teacher model and the student model and all attention heads of each layer, and extracting the attention matrices of the teacher model and the student model in the first The attention matrix of the layer, the elements of the attention matrix represent the strength of the association between two basic units in the input text.

6. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The cross-task attention alignment loss is calculated based on the teacher model and the student model, including calculating the L2 norm distance between the attention matrices of the teacher model and the student model as the local structure alignment loss. A head-by-head alignment strategy is adopted. All attention heads are selected for the multi-head attention mechanism of each layer, and alignment calculation is performed on each attention head separately. The loss of all heads is averaged. The alignment loss expression is: ， Where n represents the total number of low-resource language translation tasks, L represents the number of model intermediate layers involved in alignment, and H represents the number of attention heads in each layer. represents the teacher model in the i-th low-resource language translation task In The attention matrix of the h-th attention head in the layer, Indicates that the student model is in the i-th low-resource language translation task, The attention matrix of the h-th attention head in the layer.

7. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The logits distillation is performed based on the outputs of the teacher model and the student model, including obtaining the original outputs of the teacher model and the student model, generating a softened probability distribution based on the scaling mechanism of the original outputs based on the temperature parameter, and quantifying the difference in the softened probability distributions of the teacher and student models through KL divergence to obtain the logits distillation loss, which is expressed as: ， in, and They represent the predicted probabilities of the teacher model and the student model for the cth word, respectively, and C represents the number of vocabulary categories.

8. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The logits distillation is performed based on the outputs of the teacher model and the student model, and the task weights are calculated based on the task loss and the task average loss, and the task weights are normalized. Based on the principle of multi-loss collaborative supervision, the distillation loss is integrated with the alignment loss and the loss of each language task through the total loss function. The loss of each language task is the cross entropy loss of the student model on the low-resource language translation task. The total loss function formula is: ， in, represents a hyperparameter used to adjust the weight of distillation loss in the total loss. represents the logits distillation loss, represents the cross-task attention alignment loss, represents the hyperparameter used to adjust the weight of the cross-task attention alignment loss, represents the dynamic weight of the i-th low-resource language translation task.

9. The low-resource language translation method based on deconstructive distillation according to claim 1, characterized in that: The deployment of low-resource language translation based on the trained student model includes pre-deployment optimization of the trained student model, converting model parameters from 32-bit floating point numbers to 8-bit integers using the INT8 quantization method, and implementing the quantization process through linear mapping to convert the model into a format for end-side deployment. The deployment is completed by eliminating redundant computing nodes and fusing operators.

10. A low-resource language translation system based on deconstructive distillation, executed according to the method of claim 1, characterized in that: include: The data acquisition module is configured to: acquire parallel corpus data of low-resource languages ​​and common languages; The teacher model module is configured to: construct a teacher model using the parallel corpus data as input, including pre-training the BERT large language model using the BERT large language model as the architecture and fine-tuning the BERT large language model using all parameters to obtain the teacher model; The student model module is configured to build the student model backbone based on the pre-trained BERT large language model and optimize the student model, including lightweight processing of the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method; The alignment module is configured to calculate the cross-task attention alignment loss based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the teacher and student model attention matrices using the L2 norm; The optimization module is configured to perform logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss; The transformation module is configured to: complete the deployment of low-resource language translation based on the trained student model.

Images