A low-resource language translation method and system based on deconstructive distillation

Abstract

The application relates to the technical field of language translation, in particular to a low-resource language translation method and system based on a deconstructive distillation, the method comprising the following steps: acquiring parallel corpus data of a low-resource language and a general language; constructing a teacher model by taking the parallel corpus data as input; constructing a student model trunk based on a pre-trained BERT large language model and optimizing the student model; calculating a cross-task attention alignment loss based on the teacher model and the student model; executing logits distillation based on the teacher model and the student model output; and completing the deployment of low-resource language translation based on the trained student model. The application provides a high-quality knowledge migration source for the student model by constructing a BERT teacher model fine-tuned by all parameters, and effectively solves the translation precision problem caused by the lack of low-resource language data by means of double supervision of cross-task attention alignment and logits distillation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language translation, in particular to a low-resource language translation method and system based on deconstructive distillation. BACKGROUND

[0002] With the advancement of globalization, the demand for low-resource language translation is increasing, which has important practical value in cross-border communication, cultural heritage, education popularization and other fields. However, due to the scarcity of corpus and the lack of model training data, the performance of existing translation systems is limited. At the same time, due to the limited computing resources and storage capacity of edge devices such as portable translation terminals, large pre-training language models are difficult to deploy directly, and lightweight models are difficult to capture the complex semantic features of low-resource languages due to the limitation of parameter size, resulting in low translation accuracy and poor generalization ability.

[0003] Existing parameter-efficient fine-tuning techniques such as MoLoRA can reduce model parameters to some extent, but the lower projection matrix of the LoRA expert module has high redundancy, and the feature transformation path between multiple tasks has high coincidence, resulting in large memory occupation and long inference time, which cannot meet the lightweight needs of edge devices. Traditional knowledge distillation methods rely on logits transmission of the output layer, ignoring the semantic modeling strategy of the attention mechanism in the middle layer of the teacher model. In low-resource language translation, the grammar structure and semantic association of different languages differ significantly, and the student model is difficult to capture the deep alignment logic of the teacher model through a single output layer distillation, resulting in semantic rupture in the translation process. Low-resource language translation often involves multiple scenarios, and existing methods lack a mechanism for dynamically adjusting distillation weights, making it difficult to allocate resources according to the translation difficulty of different scenarios. At present, there is a need for a low-resource language translation method and system based on deconstructive distillation. SUMMARY

[0004] To solve the problem of inaccurate resource allocation for different scenarios and poor adaptability to edge devices in existing methods, the present application provides a low-resource language translation method and system based on deconstructive distillation.

[0005] In the first aspect, the present application provides a low-resource language translation method based on deconstructive distillation, which adopts the following technical solution:

[0006] A low-resource language translation method based on deconstructive distillation, comprising:

[0007] Obtaining parallel corpus data of low-resource languages and general languages;

[0008] Constructing a teacher model using parallel corpus data as input, including using BERT large language model as architecture, and pre-training BERT large language model using full parameter fine-tuning to obtain teacher model;

[0009] The student model trunk is constructed based on the pre-trained BERT large language model, and the student model is optimized, including lightweight processing of the BERT large language model, and optimization of the student model by using a structural deconstruction low-rank disturbance method;

[0010] The cross-task attention alignment loss is calculated based on the teacher model and the student model, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the attention matrices of the teacher model and the student model by L2 norm;

[0011] The logits distillation is performed based on the outputs of the teacher model and the student model, including introducing a temperature parameter to generate the softened probability distribution of the teacher model and the student model, and calculating the KL divergence as the distillation loss;

[0012] The low-resource language translation deployment is completed based on the trained student model.

[0013] Further, the teacher model is constructed by using parallel corpus data as input, including using a BERT large language model as a basic architecture, the BERT large language model including an L-layer Transformer encoder, each layer being composed of a multi-head self-attention mechanism and a feedforward neural network, taking low-resource language text as an input sequence and corresponding general language text as a target output sequence, the input sequence being processed into a token sequence by a tokenizer and adding a marker symbol, and being converted into a fixed-length word embedding vector sequence.

[0014] Further, the BERT large language model is pre-trained by using full-parameter fine-tuning, including obtaining context representation by encoding the word embedding vector sequence through the BERT large language model, mapping the context representation into a word probability distribution of the target language through an output layer, defining a cross-entropy loss function according to the word probability distribution, and minimizing the cross-entropy loss function by using a gradient descent algorithm, the word probability distribution expression being:

[0015] ,

[0016] wherein, represents the conditional probability distribution of the output target sequence y given the input sequence x under the condition that the teacher model parameter is represents a normalization function, represents a weight matrix of the output layer, represents the context representation obtained after encoding the input sequence x by the BERT model, represents a bias vector of the output layer.

[0017] ​Further, the student model is optimized by using the structural deconstruction low-rank disturbance method, which includes compressing the input low-resource language text features through a shared encoding matrix to generate general features, then defining a task-condition decoding path, obtaining a task embedding vector through language ID mapping, mapping the task embedding vector into an intermediate matrix using a structure regulation network, multiplying the intermediate matrix with a shared decoding kernel to generate a structure control decoding matrix, and generating an overall disturbance according to the structure control decoding matrix and the general features.

[0018] Further, the attention matrix of the corresponding layer of the teacher model and the student model is extracted, which includes selecting the intermediate layer and all attention heads of each layer of the teacher model and the student model based on all translation tasks of the parallel corpus, and extracting the attention matrix of the teacher model and the student model at the i-th low-resource language translation task.

[0019] Further, the cross-task attention alignment loss is calculated based on the teacher model and the student model, which includes calculating the L2 norm distance between the attention matrices of the teacher model and the student model as a local structure alignment loss, using a head-by-head alignment strategy, selecting all attention heads for the multi-head attention mechanism of each layer, and calculating the alignment for each attention head separately, and taking the average of the losses of all heads, the alignment loss expression is:

[0020] ,

[0021] wherein n represents the total number of low-resource language translation tasks, L represents the number of intermediate layers of the model participating in alignment, H represents the number of attention heads of each layer, represents the attention matrix of the i-th low-resource language translation task in the teacher model, the h-th attention head of the L-th layer, represents the attention matrix of the i-th low-resource language translation task in the student model,

[0022] Further, logits distillation is performed based on the outputs of the teacher model and the student model, which includes obtaining the original outputs of the teacher model and the student model, generating a softened probability distribution based on the scaling mechanism of the temperature parameter on the original output, quantifying the difference between the softened probability distributions of the teacher and student models through KL divergence to obtain logits distillation loss, and the expression is:

[0023] ,

[0024] wherein and ​respectively represent the prediction probability of the teacher model and the student model on the cth vocabulary, and C represents the number of vocabulary categories.

[0025] Further, the execution of logits distillation based on the teacher model and the student model output further includes calculating a task weight according to a task loss and a task average loss, and normalizing the task weight, based on the principle of multi-loss collaborative supervision, fusing the distillation loss and the alignment loss, and each language task loss through a total loss function, the each language task loss being a cross-entropy loss of the student model on a low-resource language translation task, and the total loss function formula being:

[0026] ,

[0027] wherein, represents a hyperparameter for adjusting the weight of the distillation loss in the total loss, represents a logits distillation loss, represents a cross-task attention alignment loss, represents a hyperparameter for adjusting the weight of the cross-task attention alignment loss, represents a dynamic weight of the ith low-resource language translation task.

[0028] Further, the deployment of the low-resource language translation based on the trained student model includes pre-deployment optimization of the trained student model, conversion of model parameters from 32-bit floating-point numbers to 8-bit integers using an INT8 quantization method, realization of the quantization process through linear mapping, conversion of the model into an end-side deployment format, and completion of the deployment by eliminating redundant calculation nodes and fusing operators.

[0029] In a second aspect, a low-resource language translation system based on deconstructive distillation includes:

[0030] A data acquisition module configured to acquire parallel corpus data of low-resource languages and general languages;

[0031] A teacher model module configured to construct a teacher model by taking the parallel corpus data as input, including taking a BERT large language model as an architecture, and pre-training the BERT large language model by full-parameter fine-tuning to obtain the teacher model;

[0032] A student model module configured to construct a student model main body based on the pre-trained BERT large language model and optimize the student model, including lightweight processing of the BERT large language model, and optimizing the student model by a structural deconstructive low-rank perturbation method;

[0033] The alignment module is configured to calculate a cross-task attention alignment loss based on the teacher model and the student model, including extracting attention matrices of corresponding layers of the teacher model and the student model, and measuring the difference between the attention matrices of the teacher and the student model through an L2 norm;

[0034] The optimization module is configured to perform logits distillation based on the teacher model and the student model output, including introducing a temperature parameter to generate soft probability distribution of the teacher model and the student model, and calculating KL divergence as a distillation loss;

[0035] The transformation module is configured to complete the deployment of low-resource language translation based on the trained student model.

[0036] To sum up, the present application has the following beneficial technical effects:

[0037] 1. The present application builds a BERT teacher model fine-tuned by full parameters, uses its deep learning ability on parallel corpus to provide high-quality knowledge transfer source for the student model, and at the same time, with the double supervision of cross-task attention alignment and logits distillation, makes the lightweight student model not only reproduce the intermediate layer semantic analysis strategy of the teacher model, but also accurately match the output layer vocabulary prediction distribution, effectively makes up for the translation precision problem caused by the lack of low-resource language data, especially in the language translation task with complex grammar and scarce data.

[0038] 2. The present application introduces dynamic task weight, and adaptively adjusts the supervision intensity according to the loss of each language task, so that the model can preferentially optimize the difficult task, improve the global adaptation ability to different low-resource languages, share the encoding matrix and decoding kernel to promote cross-language knowledge transfer, avoid information isolation between tasks, and make the model not need to be trained from scratch when a new language task is added, but only need to update the task embedding vector to quickly adapt.

[0039] 3. The student model of the present application reduces the number of layers and reduces the hidden dimension to reduce the parameter quantity, combines INT8 quantization and operator optimization, significantly reduces the memory occupation and inference delay, can be efficiently deployed on edge devices, and the structure deconstruction type low rank disturbance method further reduces the parameter redundancy, the parameter quantity growth rate is much lower than that of the traditional method in the multi-task scene, and the model efficiency and cross-language adaptability are considered. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the overall flowchart of a low-resource language translation method based on deconstruction distillation according to an embodiment of the present application.

[0041] Figure 2 is a framework diagram of a low-resource language translation method based on deconstruction distillation according to an embodiment of the present application.

[0042] Figure 3is a framework diagram of a structural deconstruction low-rank perturbation method of an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application is further described in detail below with reference to the accompanying drawings.

[0044] Embodiment 1

[0045] Referring to Figure 1 The low-resource language translation method based on deconstruction distillation of the present embodiment comprises:

[0046] Obtaining parallel corpus data of a low-resource language and a general language;

[0047] Constructing a teacher model by taking the parallel corpus data as input, including taking a BERT large language model as architecture, and pre-training the BERT large language model by full parameter fine-tuning to obtain the teacher model;

[0048] Constructing a student model backbone based on the pre-trained BERT large language model and optimizing the student model, including lightweight processing of the BERT large language model, and optimizing the student model by a structural deconstruction low-rank perturbation method;

[0049] Calculating a cross-task attention alignment loss based on the teacher model and the student model, including extracting attention matrices of corresponding layers of the teacher model and the student model, and measuring the difference between the attention matrices of the teacher and student models by L2 norm;

[0050] Performing logits distillation based on the outputs of the teacher model and the student model, including introducing a temperature parameter to generate softened probability distribution of the teacher model and the student model, and calculating KL divergence as distillation loss;

[0051] Completing deployment of low-resource language translation based on the trained student model.

[0052] Specifically, the low-resource language translation method based on deconstruction distillation comprises the following contents:

[0053] S1, obtaining parallel corpus data of a low-resource language and a general language;

[0054] As Figure 1 , Figure 2As shown, the scope of the target low-resource language is determined, which refers to a language with less labeled data in the field of natural language processing, and the general language is selected as a widely used and resource-rich language, such as Chinese and English. Secondly, parallel corpus data of the low-resource language and the general language is collected through various channels, including but not limited to public language resources, bilingual texts accumulated by professional translation agencies, and bilingual data collected by crowdsourcing platforms after manual verification. The collected parallel corpus covers daily conversations, news reports, academic literature, and life services to improve the diversity and coverage of the corpus.

[0055] Next, the collected original parallel corpus is preprocessed, including removing noise data in the text, unifying the text format, and performing sentence-level alignment. That is, by length-based comparison and lexical overlap calculation, the sentences of the low-resource language are matched one by one with their corresponding general language translation sentences to form sentence-level parallel sentence pairs. For long sentences, they are split into semantically complete short sentences to improve alignment accuracy, and for short sentences, they are kept as they are.

[0056] Finally, the processed parallel corpus is divided into training set, validation set and test set according to the preset ratio 7:2:1. The training set is used for subsequent teacher model and student model training process, the validation set is used to monitor the performance change in the model training process to adjust the hyperparameters, and the test set is used to evaluate the translation effect of the model. All corpus is stored in a structured format for subsequent steps.

[0057] S2, constructing a teacher model with parallel corpus data as input, including using BERT large language model as architecture and pre-training BERT large language model by full parameter fine-tuning to obtain the teacher model;

[0058] The low-resource language and general language parallel corpus data obtained in step S1 are used as input to construct a teacher model, which is based on a BERT large language model architecture. The BERT large language model includes L-layer Transformer encoder, L=12 layers in this embodiment, which can be selected according to the complexity of the low-resource language. Each layer of the Transformer encoder is composed of a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism includes H attention heads for capturing the association between tokens in the input sequence from different semantic dimensions. The feedforward neural network is composed of two linear transformation layers and a GELU activation function in the middle, which is used to convert and enhance the output of the attention mechanism.

[0059] In the model construction process, the low-resource language text is taken as the input sequence, and the corresponding general language text is taken as the target output sequence. First, the input sequence is processed, and the low-resource language tokenizer adapted to the BERT model is used to split the input sequence into the smallest semantic unit token to form a token sequence. Then, the special symbol [CLS] is added at the beginning of the token sequence to represent the overall semantics of the sequence, and the special symbol [SEP] is added at the end to identify the end of the sequence. The token sequence is adjusted to a fixed length, such as 512 tokens, by padding operation. If the length is insufficient, it is filled with [PAD], and if it is too long, it is truncated to the length.

[0060] The processed token sequence is then converted into a word embedding vector sequence. Each token is mapped to a corresponding word vector through a word embedding table, while incorporating position embedding to represent the position information of the token in the sequence and paragraph embedding to distinguish different language sequences. The three are superimposed to form the final word embedding vector sequence, which has a dimension of [fixed length, hidden layer dimension].

[0061] After that, the BERT large language model is pre-trained in a full-parameter fine-tuning manner. The word embedding vector sequence is input into the L-layer Transformer encoder of the BERT large language model, and the context representation H of the input sequence is obtained after layer-by-layer encoding. The processing process of each layer of the Transformer encoder is as follows: the word embedding vector sequence is first calculated by the multi-head self-attention mechanism to obtain the attention weighted feature. The calculation of the multi-head self-attention mechanism is based on the formula:

[0062]

[0063] where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, obtained by linear transformation of the input feature, is the dimension of each attention head, and softmax is a normalization function used to convert the similarity between tokens into attention weights. Then, the feature is connected in residual with the input feature and processed by layer normalization to obtain the output of the multi-head self-attention mechanism. The output is then input into the feedforward neural network, which is mapped from the hidden layer dimension to 4 times the hidden layer dimension by the first linear transformation layer, activated by the GELU activation function, and mapped back to the hidden layer dimension by the second linear transformation layer. After that, the output of the feedforward neural network is connected in residual with the input of the feedforward neural network and processed by layer normalization to obtain the output of the current Transformer layer, which is used as the input of the next layer. The context representation H output by the encoder is processed by the output layer to map it to the word probability distribution of the target language. The calculation process of the output layer is based on the formula:

[0064] ,​

[0065] wherein, denotes the conditional probability distribution of outputting target sequence y given input sequence x under the condition that the teacher model parameters are denotes the normalization function, denotes the weight matrix of the output layer, denotes the context representation obtained after input sequence x is encoded by the BERT model, denotes the bias vector of the output layer.

[0066] In the pre-training process, the cross-entropy loss function is defined according to the word probability distribution and the true label of the target output sequence, i.e., the token sequence corresponding to the general language text, and the formula is:

[0067] ,

[0068] wherein, is the one-hot vector of the true label, and the correct token corresponds to 1 and the rest is 0, denotes the set of teacher model parameters, and the difference between the model prediction probability and the true label is quantified by the formula, and then the gradient descent algorithm is used to minimize the cross-entropy loss function to iteratively update all parameters of the teacher model Each iteration uses a batch of data from the training set to update the parameters, and the BLEU score of the model is monitored on the validation set. When the BLEU score on the validation set does not improve for multiple epochs, the training is stopped, and the final trained teacher model is obtained.

[0069] S3, based on the pre-trained BERT large language model, a student model backbone is constructed and the student model is optimized, including lightweight processing of the BERT large language model and optimization of the student model using a structural deconstruction low-rank perturbation method;

[0070] As shown in Figure 3 , based on the pre-trained BERT large language model in step S2, a student model backbone is constructed and the student model is optimized. First, the BERT large language model is lightweight processed, and a BERT model of the same series as the teacher model is selected as the student model backbone. By reducing the number of layers of the Transformer encoder from 12 to 6, reducing the hidden layer dimension from 768 to 512, and reducing the number of attention heads from 12 to 6, while ensuring that the model retains the core semantic encoding ability, such as understanding the basic vocabulary and grammar structure of low-resource languages, the overall parameter quantity is reduced by about 60% to adapt to resource-constrained scenarios such as edge devices, forming the basic backbone structure of the student model. ​

[0071] Subsequently, a structural deconstruction-based low-rank perturbation method is used to optimize the student model, enhancing its adaptability to multiple low-resource language tasks and reducing parameter redundancy. The specific process is as follows: The input feature dimension is defined as n, i.e., the low-resource language text feature dimension output by the student model backbone; the output feature dimension is m, i.e., the feature dimension adapted to the general language; the perturbation rank is r, used to control the low-rank space dimension; the low-resource language task embedding dimension is k; the intermediate dimension is h; and the overall perturbation expression is represented by the core formula:

[0072] ,

[0073] in, For input features, A shared encoding matrix used for all tasks to compress input features. To activate the transformation function, which is used to improve the expressive power and stability of the projection space. Indicates task The structure-controlled decoding matrix is ​​implemented through three stages: shared compression path, task-conditional decoding path, and perturbation integration. In the shared compression path, the shared encoding matrix... Shared across all low-resource language tasks, this feature extracts general language features from the input features, and is applied to the low-resource language text features output by the student model backbone. Compress, through Generate general features after activation function processing The corresponding formula In this part, shared compression and nonlinear transformation of input features are achieved; in the task-conditional decoding path, for the i-th low-resource language translation task, a unique task embedding vector is obtained through language ID mapping, such as assigning ID=0 to Tibetan and ID=1 to Swahili. This vector encodes language-specific information, such as grammatical rules and vocabulary collocation habits, using a structural regulation network. Embedding the task vector The mapping is to an intermediate matrix. The structure control network is implemented using a two-layer fully connected network. The first layer maps the task embedding vector from dimension k to the intermediate dimension h, as shown in the formula:

[0074] ,

[0075] in, This is the first layer weight matrix. This is the first layer bias vector. As the activation function, the second layer maps the intermediate results from dimension h to dimension r, as shown in the formula. ,in, This is the weight matrix for the second layer. For the second layer bias vector, the intermediate matrix is finally obtained , and then multiplied by a shared decoding kernel , which represents the basic decoding matrix shared by all tasks, and the uniform semantic mapping rule generates the structure control decoding matrix:

[0076]

[0077] wherein, is the task embedding vector, obtained by task ID mapping, is a structure regulation network that maps the task embedding to a structure control matrix , is the decoding weight kernel shared by all tasks, that is, in the formula; this matrix integrates the task-specific information of a specific low-resource language; in the disturbance integration link, the overall disturbance is generated according to the structure control decoding matrix and the general feature, and the overall disturbance: is obtained by matrix multiplication of and , which completely corresponds to the core formula , and realizes the superposition of task-specific disturbance on the basis of general features, and the traditional LoRA module, that is, each task needs to introduce a parameter amount , and the total parameter amount is , , compared with the number of tasks, the total parameter amount of the method is calculated by the formula:

[0078]

[0079] wherein, is the parameter amount of the structure regulation network , and corresponds to the parameter amount of the encoding matrix , wherein r is the disturbance rank, controlling the low-rank space dimension, n is the input feature dimension, m is the output feature dimension, and h is the intermediate feature dimension, is the total number of low-resource language tasks, is the dimension of the embedding vector of each task, when the number of tasks N is large and , the parameter scale is significantly reduced, effectively reducing the parameter redundancy; at the same time, this design provides a continuous differentiable function mapping from the task embedding space to the disturbance matrix , which can be optimized in the training process with the backbone distillation target; the final output of the student model is the superposition of its backbone output and the overall disturbance, that is, wherein, ​​The basic result output by the student model main trunk enables the model to dynamically optimize the output for different low-resource languages; finally, the lightweight main trunk and the structural deconstruction low-rank disturbance module are combined to form a complete student model, and the main trunk parameters are inherited from the corresponding layer parameters of the pre-trained BERT during initialization, and the disturbance module parameters, including and all task embedding vectors are randomly initialized, laying a foundation for subsequent fine-tuning through attention alignment and knowledge distillation.

[0080] S4, based on the teacher model and the student model, a cross-task attention alignment loss is calculated, including extracting the attention matrices of the corresponding layers of the teacher model and the student model, and measuring the difference between the attention matrices of the teacher and the student model through L2 norm;

[0081] Based on the teacher model trained in step S2 and the student model constructed in step S3, the cross-task attention alignment loss is calculated to achieve effective migration of the intermediate layer structure-level semantic information, the specific process is as follows: first, for all low-resource language translation tasks covered in step S1 parallel corpus, the intermediate layers of the model participating in attention alignment and the attention heads of each layer are determined, the intermediate layers of the teacher model and the student model that can reflect the core semantic modeling strategy are selected, the 3rd to 8th layers of the Transformer encoder are selected in this embodiment, L=6 layers in total, and all attention heads of each layer are included, H=12 heads, to ensure that the alignment supervision covers the key semantic processing stage of the model; then, for each low-resource language translation task , the attention matrices of the lth layer and the hth attention head of the teacher model and the student model are extracted respectively, wherein the attention matrix of the teacher model is denoted as , and the attention matrix of the student model is denoted as The element values of these attention matrices represent the association strength between two tokens (basic semantic units) in the input text, and the greater the value, the closer the association, for example, in the Tibetan-Chinese translation task, the matrix element may reflect the grammatical dependency relationship between “ ” (I) and “my”; next, the local structure alignment loss is calculated using the head-by-head alignment strategy, and the alignment calculation is performed separately for each attention head, and the difference between the corresponding attention matrices of the teacher model and the student model is measured through L2 norm, and the calculation formula is , wherein represents the square of the L2 norm, which is used to quantify the overall difference between the two matrices at the element level; then, the local losses of all tasks, all participating alignment intermediate layers and all attention heads are aggregated to obtain the cross-task attention alignment loss, and the expression is:

[0082] ,

[0083] wherein n represents the total number of low-resource language translation tasks, L represents the number of intermediate layers of the model participating in alignment, and H represents the number of attention heads of each layer, represents the attention matrix of the i-th low-resource language translation task in the teacher model, the h-th attention head of the j-th layer, represents the attention matrix of the i-th low-resource language translation task in the student model, the contribution of different tasks, layers and attention heads to the total loss is balanced by normalizing the triple summation result by nL; the design of the loss function is based on the characteristics of the attention matrix in the Transformer architecture, which explicitly models the token-to-token dependency and implicitly reflects the semantic modeling preference, and by forcing the attention distribution of the student model to align with the teacher model, the problem of shallow alignment and deep deviation caused by output layer distillation is avoided, and the effective migration of the semantic modeling strategy in the intermediate layer is ensured; finally, the cross-task attention alignment loss calculated is used for total loss function fusion in subsequent step S5, which cooperates with the distillation loss and the task loss to supervise the training of the student model, so that the student model can inherit the output layer knowledge of the teacher model while simultaneously learning the structural semantic processing ability of the intermediate layer.

[0084] S5, logits distillation based on the outputs of the teacher model and the student model, including introducing a temperature parameter to generate soft probability distribution of the teacher model and the student model, and calculating KL divergence as distillation loss;

[0085] The trained teacher model and student model perform logits distillation and fuse multiple loss supervision to optimize the student model. First, after the cross-task attention alignment in step S4, the representation of the final output of the teacher model is migrated to the student model through logits distillation, forming a double knowledge transfer path of structural migration and output alignment, where the teacher model is denoted as T and the student model is denoted as S. In logits distillation, the original outputs of the teacher model and the student model are first obtained, i.e., the original logits output of the teacher model and the original logits output of the student model, the original outputs are scaled based on the temperature parameter to generate soft probability distribution, the temperature parameter τ is set to 2.0, and high temperature τ helps the teacher model T to output smoother probability distribution, thereby alleviating the gradient explosion problem in the learning process of the student model S, the soft probability distribution calculation formula of the teacher model is , and the soft probability distribution calculation formula of the student model is wherein is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, is a normalization function that converts the scaled logits into a probability distribution, <000 The original logits output, For student models The original logits output, where τ is the temperature parameter, is then used to quantify the difference in softening probability distributions between the teacher and student models using KL divergence, yielding the logits distillation loss, expressed as:

[0086] ,

[0087] in, and Let represent the prediction probabilities of the teacher model and the student model for the c-th word, respectively, where C represents the number of word categories. This loss function measures the difference between the two probability distributions, encouraging the student model to mimic the output distribution of the teacher model. To further improve the task adaptability of the student model, task-adaptive distillation weights are introduced. Specifically, let be the weights in the current t-th training round, where the task... The loss is The average loss for all tasks is The task weight update rule is defined as follows:

[0088] ,

[0089] in, It is a hyperparameter that controls the rate of weight adjustment. This represents the average loss across all tasks, when the loss of a particular task is... A value above the average indicates that the task is relatively difficult or the model performs poorly on this task; the corresponding weights are... The weights will increase if the weights decrease and decrease if the weights decrease. Furthermore, this weight update mechanism conforms to the exponential decay paradigm of change stability; when the task loss tends towards the average loss, ... ,in, This represents the limiting condition, i.e., when the loss of the i-th task... Infinitely close to the average loss of all tasks hour, This represents the updated weight of the i-th task in the (t+1)-th training round. This represents the current weight of the i-th task in the t-th training round. This indicates that the task weights tend to stabilize after convergence, avoiding drastic fluctuations in weights in the later stages of training, which is beneficial for the final global optimization. To ensure that the sum of all task weights is 1, and to prevent overfitting to local tasks, all task weights need to be normalized using the following formula:

[0090] ,

[0091] in, is the weight sum of all tasks after the t-th round of training, and by dividing the weight of each task by the sum of all task weights, the normalized weight is ensured satisfies .

[0092] Finally, based on the principle of multi-loss collaborative supervision, the distillation loss is fused with the alignment loss and the language task loss through the total loss function, and the language task loss is the cross-entropy loss of the student model on the low-resource language translation task The total loss function formula is:

[0093] ,

[0094] wherein, represents a hyperparameter for adjusting the weight of the distillation loss in the total loss, represents the logits distillation loss, represents the cross-task attention alignment loss, represents a hyperparameter for adjusting the weight of the cross-task attention alignment loss, represents the dynamic weight of the i-th low-resource language translation task, and by minimizing the total loss function, the student model can achieve better performance on the target task , realizing the collaborative optimization of structure-level alignment and output-level distillation, and improving the generalization ability of the model among multiple tasks.

[0095] S6, based on the trained student model, complete the deployment of low-resource language translation.

[0096] The trained student model, which is a lightweight model optimized by structural deconstruction low-rank perturbation and fused with multi-loss supervision, is deployed for the actual application scenario of low-resource language translation. The specific process is as follows: first, the trained student model is optimized before deployment to adapt to the resource constraints of edge devices and local servers. The optimization process includes quantization processing and format conversion. The quantization processing uses INT8 quantization technology to convert model parameters from 32-bit floating-point numbers to 8-bit integers, which can greatly reduce memory occupancy under the premise of controllable precision loss. The quantization process is realized through linear mapping. Let the original parameter be , the quantized parameter , wherein, 255 is the maximum value of 8-bit integer, the original parameter is linearly mapped to the range of 8-bit integer by the formula, and the relative distribution characteristics of the parameter are preserved; the format conversion stage converts the model into a format suitable for end-side deployment, and eliminates redundant calculation nodes and fusion operators through a static graph analysis tool, specifically including removing nodes only used for training, such as dropout layers, update nodes of BatchNorm layers, and invalid operation nodes, while fusing continuous calculation operators into a single composite operator to reduce the number of memory read / write times in the calculation process and improve inference efficiency; then, according to the hardware architecture of the deployment device, the corresponding inference engine is selected, for mobile devices with ARM architecture, TensorFlowLite or ONNXRuntimeMobile engine is adopted, and its calculation kernel optimized for low-power hardware is used.

[0097] For local servers with x86 architecture, ONNXRuntime or PyTorchJIT engine is adopted, which supports multi-thread parallel inference and meets the low-latency requirements of hundreds of learners using at the same time; then, an end-to-end low-resource language translation process is designed, the input preprocessing stage receives the low-resource language text input by the user, converts it into a token sequence through the same tokenizer as the training stage, adds special symbols “[CLS]” and “[SEP]” and converts it into a fixed-length word embedding vector, fills zero when the length is insufficient, and truncates when it is too long, the model inference stage inputs the word embedding vector into the deployed student model, calculates through the lightweight backbone and the structure-destructured low-rank perturbation module, and outputs the logits of the general language, which is converted into a word probability distribution through the softmax function; the translation generation stage generates the general language translation text using the greedy decoding strategy (selecting the highest probability word as the current output), and uses the sliding window mechanism to process long texts in blocks to ensure that the length of each block does not exceed the maximum sequence length of the model, which is set to 512 tokens in this embodiment, and finally completes offline deployment and dynamic update, packs the optimized model, inference engine, low-resource language basic vocabulary and tokenization rules to the edge device, and supports real-time translation in a network-free environment, when new parallel corpus (such as corrected translation text fed back by the user) is collected, an incremental model is generated by updating only the parameters of the structure-destructured low-rank perturbation module through the lightweight fine-tuning of steps S3-S5, and is pushed to the device through OTA to avoid high bandwidth consumption of full model update; after deployment, the translation accuracy, inference delay (single sentence translation time) and memory occupancy of the model are monitored in real time, and for abnormal situations, the corresponding corpus fine-tuning task is supplemented, or the model layer is further compressed to balance efficiency and performance, finally realizing efficient landing application of low-resource language translation model in education fairness, cross-language communication and other scenarios.

[0098] Embodiment 2

[0099] The difference between this embodiment and embodiment 1 is that this embodiment provides a specific experiment of a low-resource language translation method based on deconstruction distillation;

[0100] To implement the feasibility of a low-resource language translation method based on deconstruction distillation, this embodiment selects a representative data set GLUE, and the detailed information of the data set is as follows:

[0101] GLUE (General Language Understanding Evaluation) is a comprehensive natural language understanding (NLU) benchmark designed to evaluate the performance of language models in various language understanding tasks. This benchmark dataset covers a series of challenging tasks, including CoLA syntax acceptability judgment, SST-2 sentiment analysis, MRPC synonym sentence identification, STS-B semantic similarity evaluation, QQP duplicate question detection, MNLI multi-domain natural language inference, QNLI question natural language inference, RTE text entailment recognition, and WNLI Winograd syntax challenge. These tasks are designed to comprehensively test the reasoning, summarization, and transfer capabilities of LLM in language understanding. The GLUE benchmark provides a unified evaluation framework by aggregating the weighted scores of each task, becoming an important tool for comparing the effectiveness of different models, algorithms, and methods. See Table 1 for detailed statistical information about the dataset.

[0102] Table 1 GLUE dataset information table

[0103]

[0104] Table 1 represents the GLUE dataset information In the experiments of this embodiment, LLM: BERT and DeBERTaV3 are selected to use FFT method, LoRA method, MoLoRA method, TinyBERT method, DistilBERT method, BERT-PKD method, BERT TINY Methods and MobileBERT method eight kinds of fine-tuning strategies are evaluated using DistilBERT-base as the student model of BERT-base for pre-training and fine-tuning.

[0105] In this experiment, not only is the FFT and LoRA-based fine-tuning framework compared, but also those using knowledge distillation for model compression are compared. Next, this paper will analyze the performance of each fine-tuning method in different tasks through detailed experimental data.

[0106] (1) FFT: This is a fine-tuning method that optimizes all parameters of a pre-trained model on a specific task to adapt to the needs of a new task.

[0107] (2) LoRA: This is a method of fine-tuning large pre-trained models efficiently through low-rank matrix decomposition, significantly reducing trainable parameters.

[0108] (3) MoLoRA: This is an advanced method that extends LoRA, further improving the flexibility and efficiency of fine-tuning through the modular design of MoE. MoLoRA-10 and MoLoRA-15 correspond to having 10 and 15 experts, respectively.

[0109] (4) TinyBERT: This is a fine-tuning method that adopts a multi-stage training strategy based on knowledge distillation, optimizing deep semantic representations through distillation learning to compress the BERT model.

[0110] (5) DistilBERT: This is a fine-tuning method that selectively reduces the number of model layers through shallow distillation, while maintaining deep semantic understanding capabilities through training.

[0111] (6) BERT-PKD: This is a fine-tuning method that reduces model size while preserving semantic representation performance by passing path information between layers of the teacher model through path knowledge distillation.

[0112] (7) BERT TINY : This is a fine-tuning method that significantly reduces the demand for computing resources while preserving its performance as much as possible by reducing the number of model layers and parameter size.

[0113] (8) MobileBERT: This is a fine-tuning method that compresses the BERT model through knowledge distillation techniques, combining low-rank factorization and lightweight deep bottleneck layer design.

[0114] Table 2 Comparison of performance of different fine-tuning methods on GLUE dataset

[0115]

[0116] Table 3 Comparison of memory usage and inference time on GLUE dataset

[0117]

[0118] Table 2 and Table 3 visually display the results of the comparison experiments, where Table 2 is the comparison of performance of different fine-tuning methods on the GLUE dataset, and Table 3 is the comparison of memory usage and inference time on the GLUE dataset. The student model proposed in this embodiment, KD-MoLoRA, has a slightly lower score than LoRA, MoLoRA, and FFT on the GLUE dataset, and a higher score than other traditional KD methods.

[0119] In the comparative experiment, the KD-MoLoRA of the embodiment realizes about 96.7% of the performance of MoLoRA-10, realizes about 95.4% of the performance of MoLoRA-15, and the model compactness is increased by 15.8% and 20.9% respectively, while the inference time is shortened by 7.3% and 6.0% respectively. This result highlights the advantages of knowledge distillation. Specifically, a reasonable knowledge distillation strategy can compress the model so that the student model achieves a performance close to that of the teacher model, thereby effectively saving computing resources.

[0120] Compared with other fine-tuning methods using KD, the student model KD-MoLoRA of the embodiment exhibits a certain performance improvement. Specifically, on the GLUE multi-task dataset, the performance of KD-MoLoRA is improved by 15.7% compared with BERT TINY and 11.7% compared with DistilBERT. This is mainly due to the fact that KD-MoLoRA integrates the design of MoE, which can dynamically allocate appropriate experts through expert routing mechanism, so that the model can flexibly cope with diversified tasks and achieve better performance. In addition, KD-MoLoRA can still maintain stable performance in resource-constrained scenarios, which is mainly due to the design of its structural deconstruction low-rank perturbation, which can reduce the redundancy of similar experts when extracting general features, ensuring the training quality of the model even in resource-constrained environments.

[0121] Compared with updating all parameters of FFT, KD-MoLoRA also performs well. Specifically, on the GLUE dataset, the performance of KD-MoLoRA is about 95.7% of that of FFT, while the inference time is shortened by 14.6% and the structure compactness is increased by 76.1%. At the same time, KD-MoLoRA also realizes about 97.3% of the performance of LoRA, and the memory occupancy is reduced by 144.9MB and the inference time is shortened by 13.5%. This series of improvements is because KD-MoLoRA not only uses KD for simple model compression, but also because the core of the design of the KD-MoLoRA of the embodiment is to capture the deep semantic information of the teacher model through the cross-task attention alignment mechanism, and identify the importance difference of different tasks. Through the task-adaptive distillation optimization method, KD-MoLoRA can assign different weights according to the importance of the task, so as to obtain better performance in multi-task learning.

[0122] Embodiment 3

[0123] The difference between the embodiment and embodiment 1 is that the embodiment provides a low-resource language translation system based on deconstruction distillation;

[0124] The data acquisition module is configured to acquire parallel corpus data of a low-resource language and a general language;

[0125] The teacher model module is configured to construct a teacher model by taking the parallel corpus data as input, including taking a BERT large language model as an architecture, and pre-training the BERT large language model by full parameter fine-tuning to obtain the teacher model;

[0126] The student model module is configured to construct a student model trunk based on the pre-trained BERT large language model and optimize the student model, including performing lightweight processing on the BERT large language model, and optimizing the student model by using a structural deconstruction low-rank disturbance method;

[0127] The alignment module is configured to calculate a cross-task attention alignment loss based on the teacher model and the student model, including extracting attention matrices of corresponding layers of the teacher model and the student model, and measuring differences between the attention matrices of the teacher model and the student model by using an L2 norm;

[0128] The optimization module is configured to perform logits distillation based on outputs of the teacher model and the student model, including introducing a temperature parameter to generate softened probability distributions of the teacher model and the student model, and calculating KL divergence as a distillation loss;

[0129] The transformation module is configured to complete deployment of low-resource language translation based on the trained student model.

[0130] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, therefore: any equivalent changes made in the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A low resource language translation method based on a bottom-up approach, characterized in that, The application relates to a method for training a low-resource language translation model, comprising the following steps: acquiring parallel corpus data of a low-resource language and a general language; constructing a teacher model by taking the parallel corpus data as input, including taking a BERT large language model as an architecture, and pre-training the BERT large language model by full-parameter fine-tuning to obtain the teacher model; constructing a student model main body based on the pre-trained BERT large language model and optimizing the student model, including lightweight processing of the BERT large language model, and optimizing the student model by a structural deconstruction low-rank disturbance method; calculating a cross-task attention alignment loss based on the teacher model and the student model, including extracting attention matrices of corresponding layers of the teacher model and the student model, and measuring the difference between the attention matrices of the teacher model and the student model by an L2 norm; performing logits distillation based on the output of the teacher model and the student model, including introducing a temperature parameter to generate softened probability distribution of the teacher model and the student model, and calculating KL divergence as a distillation loss; deploying the low-resource language translation based on the trained student model; the structural deconstruction low-rank disturbance method for optimizing the student model, including compressing input low-resource language text features by a shared encoding matrix to generate general features, then defining a task condition decoding path, obtaining a task embedding vector by language ID mapping, mapping the task embedding vector into an intermediate matrix by a structure regulation network, multiplying the intermediate matrix with a shared decoding kernel to generate a structure control decoding matrix, and generating an overall disturbance according to the structure control decoding matrix and the general features.

2. The low resource language translation method based on deconstruction distillation according to claim 1, wherein, the teacher model constructed by taking the parallel corpus data as input, including taking a BERT large language model as a basic architecture, the BERT large language model comprising an L-layer Transformer encoder, each layer being composed of a multi-head self-attention mechanism and a feedforward neural network, taking low-resource language text as an input sequence and corresponding general language text as a target output sequence, processing the input sequence into a token sequence by a tokenizer and adding a marker symbol, and converting the token sequence into a fixed-length word embedding vector sequence.

3. The low resource language translation method based on deconstruction distillation according to claim 1, wherein, the full-parameter fine-tuning for pre-training the BERT large language model, including encoding the word embedding vector sequence by the BERT large language model to obtain context representation, mapping the context representation into a word probability distribution of a target language by an output layer, defining a cross-entropy loss function according to the word probability distribution, and minimizing the cross-entropy loss function by a gradient descent algorithm, and the word probability distribution expression being: ， wherein, denotes the conditional probability distribution of outputting the target sequence y given the input sequence x under the condition that the teacher model parameters are denotes a normalization function, denotes a weight matrix of the output layer, denotes a context representation obtained after encoding the input sequence x by the BERT model, denotes a bias vector of the output layer.​ 4. The low resource language translation method based on deconstructionist distillation as claimed in claim 1, wherein, The attention matrix of the corresponding layer of the teacher model and the student model is extracted, including all translation tasks based on parallel corpus, selecting the middle layer and all attention heads of each layer of the teacher model and the student model, respectively extracting the attention matrix of the first layer of the teacher model and the student model. The element of the attention matrix represents the association strength between two basic units in the input text.

5. The low resource language translation method based on deconstructionist distillation as claimed in claim 1, wherein, the cross-task attention alignment loss calculated based on the teacher model and the student model, including calculating the L2 norm distance between the attention matrices of the teacher model and the student model as a local structure alignment loss, adopting a head-by-head alignment strategy, selecting all attention heads for each layer of multi-head attention mechanism, separately performing alignment calculation for each attention head, and taking the mean of the losses of all heads, and the alignment loss expression being: ， wherein n represents the total number of low-resource language translation tasks, L represents the number of intermediate layers of the model participating in alignment, H represents the number of attention heads of each layer, represents the attention matrix of the i-th low-resource language translation task in the teacher model, the h-th attention head of the l-th layer, represents the attention matrix of the i-th low-resource language translation task in the student model, the h-th attention head of the l-th layer.

6. The low resource language translation method based on deconstructionist distillation as claimed in claim 1, wherein, The logits distillation based on the teacher model and the student model output includes: obtaining original outputs of both the teacher model and the student model, generating softened probability distribution based on a temperature parameter scaling mechanism for the original outputs, quantifying the difference between the softened probability distribution of the teacher and the student model by KL divergence, and obtaining logits distillation loss, expressed as: ， where, and respectively represent the prediction probability of the teacher model and the student model for the cth vocabulary, and C represents the number of vocabulary categories.

7. The low resource language translation method based on deconstructionist distillation as claimed in claim 1, wherein, The logits distillation based on the teacher model and the student model output further includes calculating a task weight according to the task loss and the task average loss, and normalizing the task weight, based on the principle of multi-loss collaborative supervision, fusing the distillation loss and the alignment loss and each language task loss through a total loss function, the each language task loss being a cross-entropy loss of the student model on a low-resource language translation task, and the total loss function formula being: ， wherein, denotes a hyper-parameter for adjusting the weight of the distillation loss in the total loss, denotes logits distillation loss, denotes cross-task attention alignment loss, denotes a hyper-parameter for adjusting the weight of the cross-task attention alignment loss, denotes the dynamic weight of the i-th low-resource language translation task.

8. The low resource language translation method based on deconstructionist distillation as claimed in claim 1, wherein, The deployment of the low-resource language translation based on the trained student model includes pre-deployment optimization of the trained student model, converting model parameters from 32-bit floating-point numbers to 8-bit integers using an INT8 quantization method, realizing the quantization process through linear mapping, converting the model into an end-side deployment format, and completing the deployment by eliminating redundant calculation nodes and fusing operators.

9. A low resource language translation system based on a bottom-up approach, performing the method of claim 1, wherein, It includes: A data acquisition module configured to acquire parallel corpus data of low-resource languages and general languages; A teacher model module configured to construct a teacher model using the parallel corpus data as input, including using a BERT large language model as an architecture and pre-training the BERT large language model using full-parameter fine-tuning to obtain the teacher model; A student model module configured to construct a student model main body based on the pre-trained BERT large language model and optimize the student model, including lightweight processing of the BERT large language model and optimizing the student model using a structural deconstruction low-rank perturbation method; An alignment module configured to calculate a cross-task attention alignment loss based on the teacher model and the student model, including extracting attention matrices of corresponding layers of the teacher model and the student model, and measuring the difference between the attention matrices of the teacher and the student model by L2 norm; An optimization module configured to perform logits distillation based on the teacher model and the student model output, including introducing a temperature parameter to generate softened probability distribution of the teacher model and the student model, and calculating KL divergence as a distillation loss; A transformation module configured to complete the deployment of low-resource language translation based on the trained student model.

Images