Lightweight Yi language part-of-speech tagging method based on DeBERTa + Transform + CRF

Abstract

The invention relates to the technical field of natural language processing, and discloses a lightweight Yi language part-of-speech tagging method based on DeBERTa + Transform + CRF. According to the method, a lightweight DeBERTa coding layer, a compact Transform coding layer and an optimized CRF decoding layer are fused, so that efficient modeling of Yi language morphological characteristics and long-distance dependence in a low-resource environment are realized; the lightweight DeBERTa coding layer is pre-trained by adopting a form and position combined mask, and the form feature recognition and dialect generalization ability is improved through a high-weight mask form prefix and a high-weight mask form prefix and a high-weight mask form prefix and a high-weight mask position code are reserved; the compact Transformer coding layer enhances the modeling capability on long-distance grammar dependence such as postpositional and the like, and parameters are not obviously increased; the CRF decoding layer is optimized, and illegal label errors are remarkably reduced through a global label transition probability matrix; according to the model, under the condition of only 5000 sentences of annotation data, the form recognition F1 value is increased by 8%-12%, the illegal label transfer error is reduced by 30%-40%, the cross-dialect accuracy is increased by 10%-15%, and a high-precision and light-weight feasible path is provided for practicability of Yi language NLP on resource-constrained equipment.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically a lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF; Background Technology Part-of-speech tagging, as a fundamental task in the field of natural language processing, aims to assign corresponding part-of-speech tags (such as nouns, verbs, and adjectives) to each word in a text, providing key grammatical feature support for higher-level tasks such as machine translation, information retrieval, and text classification. For ethnic minority languages ​​like Yi, which are rich in morphology and scarce in resources, the accuracy and efficiency of part-of-speech tagging directly determine the feasibility of implementing subsequent natural language processing technologies.

[0002] The dialectal differentiation of the Yi language poses a challenge to the understanding of part-of-speech tagging. This impact is not only reflected in the superficial differences in lexical forms, but also permeates the underlying logic of the part-of-speech system, making it difficult to unify tagging standards and limiting the generalization ability of the model.

[0003] From a lexical perspective, the differences in the written forms of the same concept in different dialects directly increase the ambiguity of part-of-speech labeling; taking the basic word "water" as an example, in the northern dialect (Liangshan area), it is written as " In the eastern dialect (Zhaotong area), it is written as " In the southern dialect (Honghe region), it is written as " The three not only differ in their written forms, but also in their subtle differences in their parts of speech within sentences: In northern dialects, " "Can be used independently as a noun (e.g., " "meaning "water is flowing"), can also be combined with prefixes to form measure words (" "means a bowl of water"); while in the eastern dialect " "When used as a noun in the same context, it needs to be paired with a specific classifier." Only then can the concept of quantity be expressed; this phenomenon of different forms of the same concept and different functions of the same form results in a reuse rate of less than 30% for cross-dialect annotated corpora, and annotators need to establish separate annotation rules for different dialects.

[0004] More importantly, dialectal differences lead to shifts in the grammatical functional boundaries of parts of speech; verb tense marking shows significant differentiation across dialects: Northern dialects use suffixes such as "- (Past tense); "- (The present progressive tense) clearly distinguishes tenses, and these markings are mandatory (e.g., " The past tense of "<eat>" must be written as " Southern dialects only add "-" when emphasizing the completed state of an action. In everyday expression, tense markers are often omitted. "Can mean both "eaten" and "is eating"); this grammatical constraint means that models trained on Northern dialect corpora will misclassify a large number of verbs with omitted tense markers as "non-finite verbs" when processing Southern dialects, resulting in a 15%-20% decrease in the accuracy of tense-related part-of-speech tagging.

[0005] Currently, Yi language part-of-speech tagging methods can be broadly categorized into traditional statistical methods and neural network methods, but both have significant drawbacks. Traditional statistical methods (such as HMM and CRF) rely on manually designed feature templates (e.g., the word itself; prefixes and suffixes; contextual vocabulary, etc.), achieving an F1 score of approximately 75% on small-scale corpora. However, they struggle to fully cover the complex morphological changes in Yi and are insufficient in capturing prefixes and suffixes unique to agglutinative languages. Furthermore, these methods are weak in modeling long-distance contextual dependencies and struggle in handling ambiguities related to multiple parts of speech (e.g., "words with multiple parts of speech"). The accuracy drops significantly when the word can be used as both the verb "eat" and the noun "food".

[0006] Neural network methods have improved the above problems to some extent, but there is still room for optimization; sequence models such as RNN and LSTM can capture contextual information, but they suffer from the gradient vanishing problem and have difficulty handling long sentences; while BERT-based pre-trained models can further improve the labeling accuracy (F1 value of about 88%-89%), but they have a large number of parameters and slow inference speed, making them difficult to deploy on mobile devices or embedded systems.

[0007] With the development of edge computing and mobile applications, model lightweighting has become a research hotspot. Its core lies in reducing the consumption of computing resources through structural optimization and parameter compression while ensuring performance. However, current mainstream lightweighting techniques, such as model pruning, knowledge distillation, and quantization training, are mostly applied to Chinese and English, and there are no effective solutions for minority languages ​​such as Yi. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF, which has the advantages of high standard accuracy and high tagging efficiency, thus solving the aforementioned technical problems. To achieve the above objectives, this invention provides the following technical solution: a lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF, comprising the following steps: S1: A lightweight Yi language corpus with part-of-speech tagging is constructed by using two-layer tags to annotate the Yi language. S2: Construct a three-level model consisting of a lightweight DeBERTa coding layer, a compact Transformer coding layer, and an optimized CRF decoding layer; S3: Lightweight optimization of the model through knowledge distillation, quantization training, and dynamic pruning; S4: Use the optimized model to annotate the Yi language text using a two-layer labeling system.

[0009] As a preferred technical solution of the present invention, the S1 dual-layer tag is a combination of core part-of-speech tag and morphological feature tag; wherein, the core part-of-speech tag is divided into 8 categories: noun corresponds to Noun; verb corresponds to V; adjective corresponds to A; pronoun corresponds to R; numeral corresponds to M; particle corresponds to Part; adverb corresponds to Adv; conjunction corresponds to Conj; and the morphological feature tag is divided into 4 categories: singular corresponds to S; plural corresponds to P; past tense corresponds to Past; and progressive tense corresponds to Prog.

[0010] As a preferred embodiment of the present invention, the annotation step of S1 for annotating the Yi language is as follows: S1a.1: Annotation of the Yi language; S1a.2: For each annotation statement derived from S1a.1, calculate the Cohen's Kappa coefficient, let it be K, and the specific expression for K is as follows: KA=(P_o P_e) / (1 P_e) Where KA is Cohen's Kappa coefficient; P_o is the labeling consistency ratio; and P_e is the expected random consistency ratio. S1a.3: Judge K and retain only the statements with KA≥0.85. The statements that do not meet the standard are revised by experts and re-evaluated until the threshold of KA≥0.85 is met, thus obtaining the Yi language part-of-speech tagging corpus for model training.

[0011] As a preferred technical solution of the present invention, the S2 lightweight DeBERTa encoding layer is composed of multiple stacked DeBERTa encoder modules, each module containing a decoupled self-attention unit and a feedforward neural network, and the lightweight DeBERTa encoding layer adopts residual connection and layer normalization for stable training. The decoupled self-attention unit calculates content attention and location attention by separating content features from location information; Content attention With positional attention The calculation formula is as follows: in, For content query matrix; This is the transpose of the content key matrix; This is the transpose of the position key matrix; For the attention dimension; For content attention; For positional attention; Content attention With positional attention The combined attention weight A is obtained by fusion, and its specific expression is as follows: in, To incorporate attention weights; This is the balance coefficient; For content attention; For positional attention; The feedforward neural network sequentially performs linear transformation, nonlinear activation, and Dropout operations; the linear transformation is performed on the input feature matrix. A linear transformation is performed to map the data to a higher-dimensional space. The matrix expression after the linear transformation is as follows: in, for The matrix after linear transformation; For the input feature matrix, The length of the input sequence; The weights of the first-level linear transformation, The dimension of the input feature. The dimension of the intermediate layer of the feedforward neural network; This is the bias vector for the first-level linear transformation; The nonlinear activation uses the ReLU activation function to perform the nonlinear mapping: in, for The activated matrix; for The matrix after linear transformation; For activation functions; The Dropout operation randomly masks some neurons in the nonlinear feature space; After the Dropout operation, the feedforward network performs dimensionality reduction through a second linear transformation to obtain the output features: in, The feature matrix output by the DeBERTa encoding layer. The sequence length of this matrix; for The activated matrix; The weights for the second-level linear transformation. To represent the dimension of the output features of the feedforward network; The dimension of the intermediate layer of the feedforward neural network; This is the bias vector for the second-level linear transformation; The training phase of the lightweight DeBERTa encoding layer is divided into a pre-training phase and a fine-tuning phase. The pre-training phase performs morphological and positional joint masking, masking the prefix and suffix morphological parts of Yi words and recording their positional information; when masking the morphological parts of Yi words, the masking probability formula is: in, Vocabulary in the Yi language part-of-speech tagged corpus; For vocabulary The probability of being masked; For vocabulary The probability of including morphological features is obtained through corpus statistics; The base probability of the random mask; This is the shape weighting coefficient; The pre-training phase implements an embedding sharing mechanism, which is optimized through a joint generator and discriminator. The joint loss function of the feedforward neural network in the pre-training phase is: in, This is the joint loss function of the feedforward neural network; This represents the loss in masked language modeling; This represents the loss for word replacement detection; λ is the balance coefficient. The mask language modeling loss The specific expression is as follows: in, It is the loss for masked language modeling; The expectation for the entire training corpus; For the first in the input sequence Each token location; The set of masked token positions; The original, correct token; This is the masked version of the input sequence; For the generator parameter set; For the generator in the generator parameter set Below, the predicted probability of the masked word; This means summing up all the token positions in the masked token position set C one by one; It is a logarithmic function; The replacement word detection loss The specific expression is as follows: in, It is the loss for replacement word detection; The expectation for the entire training corpus; For the first in the input sequence Each token location; The input sequence for the discriminator contains real words and replacement words; The first received by the discriminator One token; The original, correct token; The function is an indicator: 1 if the position contains a real word, and 0 otherwise. This determines the probability that the token has been replaced, as predicted by the discriminator. The parameter set for the discriminator; This means summing the values ​​at each position of the token in the input sequence. It is a logarithmic function; The pre-training phase, based on the embedding sharing mechanism, further executes the gradient decoupling embedding sharing mechanism, sharing the word embedding matrix between the generator and discriminator, resulting in the final loss function of the lightweight DeBERTa encoding layer pre-training phase, expressed as: in, The overall goal for optimization throughout the entire pre-training phase; To embed consistency constraints; It is the loss for masked language modeling; It is the loss for replacement word detection; 1 and 2 is the hyperparameter that balances the weights of the two parts; During the fine-tuning phase, the model employs a bidirectional encoding mechanism, which involves simultaneously performing forward and backward modeling on the input sequence to capture bidirectional contextual dependencies.

[0012] As a preferred technical solution of the present invention, the compact Transformer encoding layer in S2 includes multiple simplified Transformer encoding modules, each simplified Transformer encoding module includes a multi-head self-attention and bottleneck feedforward network, and the Transformer encoding layer is trained stably using Dropout and residual connections. The multi-head self-attention calculation The formula for calculating the attention points is: in, For the first The intermediate feature matrix output by each attention head; Indicates the first One point of attention; For the first A matrix of attention head values; Indicates the first Attention weight matrix for each attention head; The first Attention weight matrix for each attention head The expression is as follows: in, Indicates the first Attention weight matrix for each attention head; and The first The query matrix and key matrix of each attention head; Indicates the attention dimension; For activation functions; The first Attention weight matrix for each attention head The expression for sparsification is as follows: in, This is the original attention weight matrix; This is the sparsed attention matrix; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. is the global attention entropy threshold; i represents the position of the i-th token in the input sequence; j represents the position of the j-th token in the input sequence; The bottleneck feedforward network will output the feature matrix from the DeBERTa coding layer. The feature dimension is increased by a linear layer, then reduced by activation, and the transformation formula is: in, is the intermediate feature matrix after the first linear transformation and ReLU activation, and n is the sequence length of this matrix; , These are the weights and bias vectors for the first-level linear transformation, respectively. , These are the weights and bias vectors for the second-level linear transformation, respectively; The feature matrix output by the DeBERTa encoding layer; This is the output matrix of the second linear transformation; The bottleneck feedforward network introduces embedding sharing and discriminative optimization, as shown in the following expression: in, For the input feature representation of the embedded sharing mechanism; , These are the weights and bias vectors for the first-level linear transformation, used for dimensionality increase; , These are the weights and bias vectors for the second-layer linear transformation, used for dimensionality reduction. This is the output matrix of the second linear transformation; It is a non-linear activation function; The Transformer encoding layer employs Dropout and residual connections for stable training, and introduces low-dimensional manifold mapping and sparse attention constraint mechanisms. The bottleneck feedforward network uses nonlinear feature mapping. in, This is the feature representation after two-level linear transformation and nonlinear mapping; The feature matrix output by the DeBERTa encoding layer; It is a non-linear activation function; , These are the weights and bias vectors for the first-level linear transformation, used for dimensionality increase; , These are the weights and bias vectors for the second-layer linear transformation, used for dimensionality reduction. The output matrix of the second linear transformation Introducing the manifold regularization term, the formula for the manifold regularization term is: in, For manifold regularization; , For the first The first sample and the first High-dimensional feature representation of each sample; The distance is Euclidean. Similarity weights; , Indicates the first... The and the first Index of each sample; It is a norm 2; This indicates that the summation is performed on all samples in the input sequence.

[0013] As a preferred technical solution of the present invention, the optimized CRF decoding layer in S2 includes a linear projection layer and a conditional random field layer, and the CRF decoding layer is trained and inferred. The linear projection layer maps 256-dimensional features to the label space, generating an emission score matrix, the expression of which is as follows: in, For the emission fraction matrix, The number of samples in a batch. This refers to the sequence length; The linear projection weight matrix; It is the bias vector; For the input feature representation of the embedded sharing mechanism; The conditional random field layer is trained using a label transition matrix. Model the dependency between labels, assign learnable parameters only to high-frequency transition rules, and assign penalty values ​​to all low-frequency transition paths other than high-frequency transition rules; The loss calculation during the CRF layer training phase, combined with the high-frequency transition rule, is expressed as follows: in, is the loss function of the CRF layer; N is the number of training samples, and c represents the c-th training sample in the training set; The input sequence corresponding to the c-th training sample; To and The corresponding label sequence; For a CRF decoding layer in a given input sequence Under the condition, for the tag sequence The conditional probability; log is the natural logarithm; Indicates to Summing the training samples; in The expression is as follows: in, For a CRF decoding layer in a given input sequence Under the condition, for the tag sequence The conditional probability; For position Corresponding tags Launch fraction; Indicates from the label Transition to tags The score; only when the transfer is to Participate in learning at any time, A set of high-frequency transfer rules; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. Represents the first in the sequence One location; It is an exponential function; Will The total score of the entire sequence is obtained by summing the launch score and transfer score at each position; To traverse all possible label paths Calculate the exponent of its score; Represents all possible combinations of label sequences; express In position Tags; For position Corresponding tags Launch fraction; Indicates from the label Transition to tags The score; The training inference includes a training phase and an inference phase. The training phase requires calculating the joint probability of each label sequence, the specific expression of which is as follows: in, For joint probability; As a normalization factor, the loss function takes the negative log-likelihood; Emission fraction generated for the linear projection layer; The transition score in the label transition matrix represents the transition score from the previous label. Transition to current tag The score; Represents the first in the sequence The location index of each token; To and The corresponding label sequence; Indicates position Tags; To obtain the total score for the complete label sequence by summing the scores at each position; ) is an exponential function; In the inference phase, the Viterbi algorithm is used to search for the optimal label path and obtain the final part-of-speech label for each word. In the parameter optimization of the conditional random field layer, L1 sparse regularization constraints are introduced. By penalizing the sum of the absolute values ​​of the parameters, the conditional probability is defined as: in, For conditional probability; Normalization factor; For the first One characteristic function; For all time steps With characteristic function Summation; These are the weights learned from the linear projection layer and the transition matrix; It is the characteristic function; It is an exponential function; The tag for the previous time step; This is the label for that time step; For time steps Input token; And add a sparse regularization term: in, For sparse regularization terms; For sparse constraint terms; For smoothing constraint terms; and These are the hyperparameters of the sparse regularization term; For the first The input sequence of one sample; The label sequence corresponding to the input sequence; It is a logarithmic function; For model parameters The conditional probability under the following conditions; Represents the first in the training set One sample; These are model parameters; It is a norm; It is a norm 2; To Summation is performed on all samples.

[0014] As a preferred embodiment of the present invention, the knowledge distillation described in S3 uses the baseline model of the DeBERTa encoding layer without pruning or quantization as the teacher model to guide the lightweight model's learning. The lightweight model serves as the student model. The quantization training described in S3 employs INT8 quantization. The dynamic pruning described in S3 removes attention heads and neurons that contribute 30-50% of the total. The total loss of the student model is a weighted sum of the KL divergence loss and the CRF layer loss. in, The total loss for the student model; For KL divergence loss; For CRF layer loss; KL divergence loss in the formula for: in, For KL divergence loss; For student models in position For tags The probability distribution; For the teacher model in position For tags The probability distribution; This represents the total number of tags; Indicates the label in the sequence; It is a logarithmic function; Indicates to Summation of the labels; Indicates the sequence Summing the positions.

[0015] Compared with existing technologies, the lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF proposed in this invention can effectively overcome many shortcomings of existing technologies in Yi language processing, and has the following significant advantages and beneficial effects: 1. This invention introduces a morphological position joint mask pre-training mechanism, enabling the model to automatically learn morphological rules such as prefixes and suffixes, thereby strengthening the joint representation ability of morphological features and grammatical functions and fundamentally reducing the reliance on manual feature design.

[0016] 2. This invention separates semantic content from positional information through DeBERTa's decoupled attention mechanism. Combined with the multi-head attention and bottleneck structure of the compact Transformer, it effectively enhances the model's ability to capture long-distance grammatical dependencies, especially in handling phenomena such as postposition of attributive clauses and cross-sentence association between subject and object in Yi syntax.

[0017] 3. This invention achieves model lightweighting while ensuring performance through multi-stage compression optimization, including parameter pruning, quantization, and knowledge distillation. The number of model parameters is reduced from hundreds of megabytes in traditional BERT to tens of megabytes, significantly improving inference speed and enabling stable operation on resource-constrained mobile devices or embedded systems.

[0018] 4. This invention introduces a conditional random field structure into the output layer and optimizes the sequence labeling results by using global label transition probability constraints, which significantly reduces the illegal transition rate and improves the consistency and overall accuracy of part-of-speech tagging. Attached Figure Description

[0019] Figure 1 A diagram illustrating a lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF; Figure 2 This is a diagram of the core structure of the model; Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Please see Figures 1-2 A lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF includes the following steps: S1: A lightweight Yi language corpus with part-of-speech tagging is constructed by using two-layer tags to annotate the Yi language. The S1 dual-layer tag is a combination of core part-of-speech tags and morphological feature tags. The core part-of-speech tags are divided into 8 categories: nouns (Noun); verbs (V); adjectives (A); pronouns (R); numerals (M); particles (Part); adverbs (Adv); and conjunctions (Conj). The morphological features are divided into 4 categories: singular (S); plural (P); past tense (Past); and progressive tense (Prog). The steps for annotating the Yi language in S1 are as follows: S1a.1: During the annotation process, two annotators familiar with the Yi dialect and grammar completed the annotation independently, and then an expert with a linguistics background reviewed any discrepancies in the annotations. S1a.2: For each annotation statement derived from S1a.1, calculate the Cohen's Kappa coefficient, let it be K, and the specific expression for K is as follows: KA=(P_o P_e) / (1 P_e) Where KA is Cohen's Kappa coefficient; P_o is the labeling consistency ratio; and P_e is the expected random consistency ratio. S1a.3: Judge K, and only retain sentences with KA≥0.85. Sentences that do not meet the standard are revised by experts and re-evaluated until the threshold of KA≥0.85 is met, and the Yi language part-of-speech tagging corpus is obtained for model training. S2: Construct a three-level model consisting of a lightweight DeBERTa coding layer, a compact Transformer coding layer, and an optimized CRF decoding layer; The S2 lightweight DeBERTa encoder layer consists of six stacked DeBERTa encoder modules, each containing a decoupled self-attention unit; a feedforward neural network; the lightweight DeBERTa encoder layer uses residual connections and layer normalization for stable training. Decouple the self-attention units and calculate content attention separately. With positional attention Each is generated by an independent linear transformation module; Representing the semantic features of words It represents the relative and absolute positional information in the sentence; both are directly output by the decoupled attention unit and are weighted and combined through bilinear mapping in the attention calculation stage, thereby realizing the separation modeling of semantic and positional information; and are fused to obtain the comprehensive attention weight A; Content attention With positional attention The calculation formula is as follows: in, For content query matrix; This is the transpose of the content key matrix; This is the transpose of the position key matrix; For the attention dimension; For content attention; For positional attention; Content attention With positional attention The combined attention weight A is obtained by fusion, and its specific expression is as follows: in, To incorporate attention weights; This is the balance coefficient; For content attention; For positional attention; The feedforward neural network performs linear transformation, nonlinear activation, and Dropout operations sequentially; the linear transformation is performed on the input feature matrix. Perform a linear transformation to map to a higher-dimensional space to enhance the special properties. The matrix expression after linear transformation is as follows: in, for The matrix after linear transformation; The input feature matrix corresponds to the output feature matrix of the upper layer, i.e., the output of the lightweight DeBERTa or compact Transformer layer. The weights of the first-level linear transformation, The dimension of the input features, i.e., the vector dimension at each time step. The dimension of the intermediate layer of a feedforward neural network is typically greater than 1. , used to enhance nonlinear expressive power; The bias vector for the first-level linear transformation is used to translate the linear transformation result, thereby enhancing the model's expressive power and nonlinear fitting effect; nonlinear activation is performed using the ReLU activation function to execute nonlinear mapping. in, for The activated matrix; for The matrix after linear transformation; ReLU activation is employed to capture the nonlinear relationships between features and enhance the model's semantic expressiveness. Next, a Dropout operation is performed with a dropout rate of 0.1–0.3, randomly masking some neurons in the nonlinear feature space to prevent overfitting and improve model robustness. After the Dropout operation, the feedforward network undergoes a second linear transformation to reduce dimensionality, yielding the output features. in, The feature matrix output by the DeBERTa encoding layer. This represents the sequence length of the matrix, which is the number of tokens in the input sentence. for The activated matrix; The weights for the second-level linear transformation. To represent the dimension of the output features of the feedforward network, which is also the hidden dimension of the compact Transformer, The dimension of the intermediate layer of the feedforward neural network; This is the bias vector for the second-level linear transformation, used to shift and correct the results of the linear transformation. To further improve learning efficiency, this layer introduces gradient clipping and layer normalization mechanisms during the training phase to ensure numerical stability and gradient distribution balance during multiple training batches. The training of the lightweight DeBERTa encoder layer is divided into a pre-training phase and a fine-tuning phase. During the pre-training phase, morphological and positional joint masking is first performed. Prefixes and suffixes of Yi words are masked, and their positional information is recorded. When masking the morphological parts of Yi words, the masking probability formula is: in, Vocabulary in the Yi language part-of-speech tagged corpus; For vocabulary The probability of being masked, such as the prefix " -"; obtained through corpus statistics, with a value range of [0.6, 0.9]; The base probability of the random mask is fixed at 0.15; The morphological weight coefficient is set to 0.8 to enhance the priority of the morphological mask. For vocabulary The probability of including morphological features is obtained through corpus statistics; During the pre-training phase, the model is based on the morphological feature probabilities of words. and random mask probability Calculate the overall probability of it being masked. Furthermore, a masking operation is performed on some words to achieve joint modeling of morphological changes and positional information; During the pre-training phase, an embedding sharing mechanism is implemented. This mechanism optimizes the generator and discriminator together. The generator is responsible for the Masked Language Modeling (MLM) task, predicting words at masked positions and minimizing the loss. The discriminator is responsible for the Replacement Detection (RTD) task, determining whether each position has been replaced. To further improve the feature sharing efficiency and task collaboration capability during the pre-training phase, this invention introduces an embedding sharing mechanism and a joint loss optimization strategy. Through the Gradient-Disentangled EmbeddingSharing (GDES) mechanism, the basic embedding matrix is ​​shared between the generator and discriminator, and gradient decoupling is used to ensure that the parameters of each task are updated independently, thereby achieving parameter reuse and feature consistency. The joint loss function of the feedforward neural network during the pre-training phase is: in, This is the joint loss function of the feedforward neural network; This represents the loss for masked language modeling. The loss is represented by substitution detection, and λ is the balance coefficient. During the joint optimization process, the semantic recovery capability and substitution discrimination capability are improved simultaneously, realizing deep modeling of Yi language morphological features and semantic dependencies, and ultimately improving the accuracy of part-of-speech tagging and dialect generalization capability. Masked language modeling loss The specific expression is as follows: in, It is the loss for masked language modeling; The expectation for the entire training corpus; For the first in the input sequence Each token location; The set of masked token positions; The original, correct token; This is the masked version of the input sequence; For the generator parameter set; For the generator in parameters Below, the predicted probability of the masked word; This means summing up all the token positions in the masked token position set C one by one, which is used to accumulate the predicted log probability loss of the model at each masked position, and obtain the total MLM loss of the whole sentence; It is a logarithmic function; Replacement word detection loss The specific expression is as follows: in, It is the loss for replacement word detection; The expectation for the entire training corpus; For the first in the input sequence Each token location; The input sequence for the discriminator contains real words and replacement words; The first received by the discriminator One token; The original, correct token; The function is an indicator: 1 if the position contains a real word, and 0 otherwise. This determines the probability that the token has been replaced, as predicted by the discriminator. The parameter set for the discriminator; This means summing up the results at each token position in the input sequence, i.e., calculating the cumulative loss for the replacement detection prediction of each token, and finally obtaining the RTD loss of the entire sentence; It is a logarithmic function; This improvement significantly reduces the risk of overfitting while maintaining the model's semantic capture capabilities and enhances its morphological structure recognition capabilities; Building upon the joint pre-training task of DeBERTaV3, a gradient decoupling embedding sharing mechanism (GDES) is further introduced to share the word embedding matrix between the generator and discriminator, and to achieve embedding consistency through an embedding consistency constraint term. Control the consistency of gradient update direction to prevent gradient conflicts and overfitting; Based on the embedding sharing mechanism, a gradient decoupling embedding sharing mechanism is then executed to share the word embedding matrix between the generator and the discriminator, resulting in the final loss function for the lightweight DeBERTa encoding layer pre-training stage, expressed as: in, The overall goal of optimization for the entire pre-training phase is a weighted sum of the losses from multiple sub-tasks; To embed consistency constraints; It is the loss for masked language modeling; It is the loss for replacement word detection; 1 and 2 is the hyperparameter that balances the weights of the two parts; The fine-tuning phase employs bidirectional encoding: the input sequence is modeled simultaneously from forward and backward to capture contextual dependencies. In the fine-tuning phase, the model adopts a bidirectional encoding mechanism, that is, the input sequence is modeled simultaneously from forward and backward to capture bidirectional contextual dependencies. This strategy can improve the modeling ability of long sentences and cross-phrase dependencies in Yi language and ensure the contextual consistency between morphological features and syntactic relationships. The S2 compact Transformer encoding layer contains 2 to 3 simplified Transformer encoding modules. Each simplified Transformer encoding module contains a multi-head self-attention and bottleneck feedforward network. The Transformer encoding layer is trained stably using Dropout and residual connections. Multi-head self-attention, calculating the first The formula for calculating the attention points is: in, For the first The intermediate feature matrix output by each attention head participates in the construction of the final layer output through concatenation or weighted summarization, and is the core intermediate variable for multi-head attention computation; Indicates the first One point of attention; For the first A matrix of attention head values; Indicates the first Attention weight matrix for each attention head; No. Attention weight matrix for each attention head The expression is as follows: in, Indicates the first Attention weight matrix for each attention head; and The first The query matrix and key matrix of each attention head; Indicates the attention dimension; superscript It is the transpose matrix; For activation functions; No. Attention weight matrix for each attention head The expression for sparsification is as follows: in, This is the original attention weight matrix; This is the sparsed attention matrix; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. is the global attention entropy threshold; i represents the position of the i-th token in the input sequence; j represents the position of the j-th token in the input sequence; The bottleneck feedforward network outputs the feature matrix from the DeBERTa encoding layer. The feature dimension is increased through a linear layer, then reduced after activation. The feature dimension transformation formula using a compact Transformer is as follows: in, is the intermediate feature matrix after the first linear transformation and ReLU activation, and n is the sequence length of this matrix; , These are the weights and bias vectors for the first-level linear transformation, respectively. , These are the weights and bias vectors for the second-level linear transformation, respectively; The feature matrix output by the DeBERTa encoding layer; This is the output matrix of the second linear transformation; The specific expression for introducing embedding sharing and discriminative optimization is as follows: in, For the input feature representation of the embedded sharing mechanism; , These are the weights and bias vectors for the first-level linear transformation, used for dimensionality increase; , These are the weights and bias vectors for the second-layer linear transformation, used for dimensionality reduction. The feature matrix output by DeBERTa; This is the output matrix of the second linear transformation; It is a non-linear activation function; The Transformer encoding layer uses Dropout and residual connections for stable training. It also incorporates low-dimensional manifold mapping and sparse attention constraints. The bottleneck feedforward network uses non-linear feature mapping. in, This is a feature representation after two-layer linear transformation and nonlinear mapping, used for subsequent embedding sharing optimization and sequence labeling tasks; The feature matrix output by the DeBERTa encoding layer; It is a non-linear activation function; , These are the weights and bias vectors for the first-level linear transformation, used for dimensionality increase; , These are the weights and bias vectors for the second-layer linear transformation, used for dimensionality reduction. It is used only during the pre-training phase to embed shared and consistency constraints, and its output It will not participate in subsequent sequence modeling and label prediction; This is a bottleneck feedforward structure in the backbone network of the model, used to perform nonlinear transformation and dimensionality compression on the encoded features, and as input to the subsequent Transformer encoding layer or CRF decoding layer; the two are functionally independent and do not share parameters. Output matrix of the second linear transformation Introducing the manifold regularization term, the formula for the manifold regularization term is: in, For manifold regularization; , For the first The first sample and the first High-dimensional feature representation of each sample; The distance is Euclidean. Similarity weights; , Indicates the first... The and the first Index of each sample; It is a norm 2; This indicates summing over all samples in the input sequence; The S2 optimized CRF decoding layer includes a linear projection layer and a conditional random field layer, and performs training and inference on the CRF decoding layer. The linear projection layer maps 256-dimensional features to the label space, generating an emission score matrix, which is expressed as follows: in, For the emission fraction matrix, The number of samples in a batch. This refers to the sequence length, which is the number of tokens in each input sentence; The linear projection weight matrix; It is the bias vector; For the input feature representation of the embedded sharing mechanism; During training of conditional random field layers, the label transition matrix is ​​used. This matrix is ​​learned to model inter-label dependencies. Learnable parameters are assigned only to high-frequency transfer rules, and penalty values ​​are assigned to all low-frequency transfer paths outside of high-frequency transfer rules. The loss calculation during the CRF layer training phase, combined with the high-frequency transfer rules, is as follows: in, is the loss function of the CRF layer; N is the number of training samples, and c represents the c-th training sample in the training set; The input sequence corresponding to the c-th training sample; To and The corresponding label sequence; For a CRF decoding layer in a given input sequence Under the condition, for the tag sequence The conditional probability; log is the natural logarithm; Indicates to Summing the training samples; in The expression is as follows: in, For a CRF decoding layer in a given input sequence Under the condition, for the tag sequence The conditional probability; Indicates from the label Transition to tags The score directly reflects the dependency relationship between tags; only when the transfer affects the tag... Participate in learning at any time, A set of high-frequency transfer rules, whose value is fixed at -10.0; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. Represents the first in the sequence There are several positions, from 1 to T; The exponential function maps linear scores to non-negative values, allowing the model to be normalized probabilistically. Will The total score of the entire sequence is obtained by summing the launch score and transfer score at each position; To traverse all possible label paths Calculate the exponent of its score; Represents all possible combinations of label sequences; express In position Tags; For position Corresponding tags Launch fraction; Indicates from the label Transition to tags The scores directly reflect the dependencies between tags; Training inference includes a training phase and an inference phase. The training phase requires calculating the joint probability of each label sequence, the specific expression of which is as follows: in, For joint probability; As a normalization factor, the loss function takes the negative log-likelihood; Emission fraction generated for the linear projection layer; The transition score in the label transition matrix represents the transition score from the previous label. Transition to current tag The score; Represents the first in the sequence The location index of each token; To and The corresponding label sequence; Indicates position The tags are used to calculate dependencies between tags; To obtain the total score for the complete label sequence by summing the scores at each position; The function is an exponential function, which maps linear scores to non-negative stable values, making probability calculations more stable. In these two formulas, the conditional probability of a single sentence This indicates that the model performs well under a given input sequence. Generate target label sequence The probability is determined by both the launch fraction and the tag transfer fraction. During the training phase, the model calculates the joint probability of all samples. To maximize this, the loss function takes the form of negative log-likelihood: this objective function achieves global optimization of the model parameters by minimizing the negative log-likelihood, enabling the conditional random field layer to learn the optimal label transition relationship and sequence dependency features; The Viterbi algorithm is used in the inference phase to search from The optimal label path is searched to obtain the final part-of-speech label for each word. In the parameter optimization of the conditional random field layer, an L1 sparse regularization constraint is introduced. By penalizing the sum of the absolute values ​​of the parameters, the model weight matrix is ​​made sparser, thereby reducing invalid connections and improving the model's generalization ability and robustness. The conditional probability is defined as: in, For conditional probability; Normalization factor; For the first One characteristic function; For all time steps With characteristic function Summation; These are the weights learned from the linear projection layer and the transition matrix; It is the characteristic function; It is an exponential function; The tag for the previous time step; This is the label for that time step; For time step Input token; And add a sparse regularization term: in, For sparse regularization terms; For sparse constraint terms; For smoothing constraint terms; and These are the hyperparameters of the sparse regularization term; For the first The input sequence of one sample; The label sequence corresponding to the input sequence; It is a logarithmic function; For model parameters The conditional probability under the following conditions; Represents the first in the training set One sample; These are model parameters used to calculate the conditional probability of each token under each tag; It is a norm; It is a 2-norm; The input to the lightweight DeBERTa encoding layer of the three-level model in S2 is sentences from the lightweight Yi language part-of-speech tagging corpus generated by S1, including word vector matrices. and attention mask The output is a feature tensor. The input to the compact Transformer coding layer is the output of the lightweight DeBERTa coding layer. and The output is The input to the CRF decoding layer is optimized to be the enhanced feature matrix output by a compact Transformer. The set of Yi language part-of-speech tags is used as the output, which is the optimal tag sequence. ; Word vector matrix Based on a pre-trained Yi language word embedding model, each word is mapped to a 128-dimensional vector, forming a shape of... tensor; Attention mask The sequence is standardized to a length of 512, with valid words marked as 1 and filler words as 0, forming a shape of... The mask matrix; Both serve as inputs to the lightweight DeBERTa encoding layer, where the word vector matrix provides semantic and morphological information, and the attention mask controls the model's focus, effectively avoiding computational redundancy caused by invalid padding. This represents the fused semantic and morphological vector features of each word, with the shape being... ; , shape is ; , shape is ; in, Indicates the number of samples in a batch; It represents the sequence length, i.e. the number of tokens contained in the sequence, including semantic, morphological, and positional information after context fusion; In S3, knowledge distillation uses the baseline DeBERTa encoding layer model (without pruning or quantization) as the teacher model to guide the lightweight model's learning. The lightweight model is the student model. Quantization training in S3 uses INT8 quantization. Dynamic pruning in S3 removes attention heads and neurons that contribute 30-50% of the data. The total loss of the student model is a weighted sum of the KL divergence loss and the CRF layer loss. in, The total loss for the student model; For KL divergence loss; For CRF layer loss; KL divergence loss in the formula for: in, For KL divergence loss; For student models in position For tags The probability distribution; For the teacher model in position For tags The probability distribution; This represents the total number of tags; Indicates the label in the sequence; It is a logarithmic function; Indicates to Summation of the labels; Indicates the sequence Sum of the positions; S4: Use the optimized model to annotate the Yi language text using a two-layer labeling system.

[0020] Specific implementation examples and comparative experiments are as follows: This embodiment details the implementation process of a lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF, including four stages: corpus construction, model training, optimization and deployment, and performance evaluation. Experimental data shows that the method achieves an average F1 score of 88.1% on texts from the three major Yi dialect regions, with a model size of only 12.5MB, making it suitable for deployment on mobile terminals.

[0021] A two-layer labeling system is adopted, and the specific labels and their distribution are shown in Table 1: Table 1. Yi language part-of-speech tagging system and corpus distribution Table 1 shows the Yi language part-of-speech tagging system used in this invention and the distribution of various tags in the corpus.

[0022] To balance grammatical function and morphological features, this method constructs a two-layer labeling system. The core parts of speech include six categories: nouns, verbs, adjectives, pronouns, numerals, and particles. Morphological features cover four categories: singular, plural, past tense, and progressive tense. In terms of corpus distribution, nouns account for the highest proportion, approximately 40%, with singular and plural nouns accounting for 28.7% and 12.3%, respectively. Verbs are the next most common, with past and progressive tenses accounting for 29.8% combined. The distribution of adjectives, pronouns, numerals, and particles is relatively balanced, accounting for 9.5%, 5.8%, 4.1%, and 9.8%, respectively. This distribution reflects the high-frequency characteristics of nouns and verbs in the Yi language structure, while also demonstrating the rich morphological changes and frequent occurrence of morphological markers characteristic of an agglutinative language, providing statistical basis for subsequent models in morphological recognition and grammatical modeling.

[0023] This table illustrates that the designed tagging system covers the core lexical phenomena of the Yi language, can comprehensively reflect the structural features of corpora from different dialect areas, and provides a unified and scalable tagging standard for part-of-speech tagging models.

[0024] A lightweight modification based on the DeBERTa-base architecture was performed, with specific parameters shown in Table 2. Table 2 Comparison of parameters between lightweight DeBERTa and the original model Table 2 presents a comparison of key structural parameters between the lightweight DeBERTa and the original DeBERTa-base model, illustrating the specific improvements made in model structure compression and parameter optimization in this invention. By proportionally reducing the hidden layer dimension, the number of encoder layers, and the number of attention heads, the model achieves approximately 50% structural simplification while maintaining its core semantic modeling capabilities.

[0025] Specifically, the hidden layer dimension was reduced from 768 to 384, the number of encoder layers was reduced from 12 to 6, the number of attention heads was reduced from 12 to 6, the total number of parameters was reduced from 138M to 34M, and the overall compression rate reached 75.4%.

[0026] In addition, the size of the pre-training corpus was reduced from 16GB of text in the general language model to 4GB of Yi language corpus. Combined with the domain-adaptive training strategy, this not only ensured semantic expressiveness but also significantly reduced the computational resource requirements for the training and inference processes.

[0027] The comparison results in Table 2 show that the lightweight design of this invention achieves a balance between parameter reduction and performance maintenance, providing a feasible solution for low-resource training and rapid deployment of minority language models.

[0028] Innovation in pre-training tasks: Basic tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP); Special Task: Joint morphological and positional masking. The masking probability formula for the morphological part (prefix / suffix) of Yi language is as follows: in, For vocabulary The probability of including morphological features (corpus statistics) is set to 0.8 in this study. This value can effectively improve the model's ability to perceive morphological boundaries, while keeping the overall masking ratio below 30% to ensure the stability of semantic reconstruction.

[0029] The performance comparison results on the test set (10,000 sentences) are shown in Table 3: Table 3 Comparison of Yi language part-of-speech tagging performance of different methods Table 3 shows the performance comparison of the method of the present invention with various baseline models on the Yi language part-of-speech tagging task, and comprehensively evaluates parameter scale, model size, inference speed and tagging accuracy.

[0030] The results show that although traditional CRF-based statistical methods are computationally fast, their average F1 score is only 75.6%, indicating poor performance in complex morphology recognition. The BERT+CRF model shows a significant improvement in semantic expression, but due to its large model parameters (110M), its inference speed is slow, and its ability to recognize morphological features remains limited. The full DeBERT+CRF model shows a slight improvement in F1 score, but its computational cost is higher, and its inference time increases significantly.

[0031] In contrast, the lightweight DeBERTa+Transformer+CRF method proposed in this invention significantly reduces computational overhead while maintaining high accuracy. The model has only 42M parameters, 12.5MB of storage space, and the single-sentence inference time is reduced to 8.2 milliseconds. The morphological feature F1 score reaches 86.8%, which is 2.5 percentage points higher than the full model.

[0032] The results show that the method achieves the optimal balance between performance and efficiency, and has the combined advantages of high accuracy, high-speed inference and lightweight deployment, providing a generalizable technical path for natural language processing of ethnic minority languages ​​such as Yi.

[0033] A tagging and corpus system adapted to the characteristics of the Yi language was developed: a two-layer tagging system was designed, such as nouns divided into singular and plural and verbs into past and progressive tenses. A multi-domain corpus of 10,000 sentences from three major dialect areas was collected, which needs to meet the requirement of Kappa≥0.85, laying a data foundation for high-precision annotation across dialects.

[0034] The three-pronged lightweight strategy for cost reduction and efficiency improvement: knowledge distillation and teacher-guided model improvement increased the F1 score in low-resource scenarios by 8.3%; INT8 quantization reduced storage by 75%; dynamic pruning removed 30% of redundant attention heads while retaining 98.7% of performance, reducing the number of model parameters from 138M to 42M and compressing the model size to 12.5MB. The results are shown in Table 4.

[0035] Table 4 Lightweighting Strategy and Model Compression Effect This table illustrates the core strategies of this invention for model compression and efficiency optimization. Through collaborative mechanisms such as knowledge distillation, quantization, and pruning, the model achieves cost reduction and efficiency improvement, maintaining high recognition performance and generalization ability while significantly reducing storage and computation costs.

[0036] Five key advantages that break through traditional methods: an average F1 score of 88.1% with an accuracy loss of <1.4%; a morphological feature F1 score of 86.8%, surpassing traditional pre-trained models; a 7.5%-9.2% improvement in F1 score across dialects; and a single-sentence inference time of 8.2ms, adaptable to mobile terminals, as shown in Tables 5 and 6. This provides a feasible solution for edge deployment of Yi language NLP.

[0037] Table 5. Model Performance Comparison and Accuracy Retention Table 5 lists the performance of the lightweight model on key performance indicators. The results show that while significantly reducing the model size and computational cost, the method of this invention maintains an accuracy level close to that of the full model and surpasses it in morphological feature recognition.

[0038] Table 6. Model Cross-Dialect and Task Generalization Performance Table 6 shows the model's generalization ability under different dialects and corpora. The results show that the method of this invention achieves high part-of-speech tagging accuracy in the three major dialect regions of the North, East, and South, with an average F1 score improvement of 8.3% across dialects, verifying the model's robust adaptability and transferability to dialect differences.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF, characterized by: Includes the following steps: S1: A lightweight Yi language corpus with part-of-speech tagging is constructed by using two-layer tags to annotate the Yi language. S2: Construct a three-level model consisting of a lightweight DeBERTa coding layer, a compact Transformer coding layer, and an optimized CRF decoding layer; S3: Lightweight optimization of the model through knowledge distillation, quantization training, and dynamic pruning; S4: Use the optimized model to annotate the Yi language text using a two-layer labeling system.

2. The lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF as described in claim 1, characterized in that: The S1 dual-layer label is a combination of core part-of-speech labels and morphological feature labels. The core part-of-speech labels are divided into 8 categories: nouns (Noun), verbs (V), adjectives (A), pronouns (R), numerals (M), particles (Part), adverbs (Adv), and conjunctions (Conj). The morphological features are divided into 4 categories: singular (S), plural (P), past tense (Past), and progressive tense (Prog).

3. The lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF as described in claim 2, characterized in that: The annotation steps for the Yi language in S1 are as follows: S1a.1: Annotation of the Yi language; S1a.2: For each annotation statement derived from S1a.1, calculate the Cohen's Kappa coefficient, let it be K, and the specific expression for K is as follows: KA=(P_o P_e) / (1 P_e) Where KA is Cohen's Kappa coefficient; P_o is the labeling consistency ratio; and P_e is the expected random consistency ratio. S1a.3: Judge K and retain only the statements with KA≥0.

85. The statements that do not meet the standard are revised by experts and re-evaluated until the threshold of KA≥0.85 is met, thus obtaining the Yi language part-of-speech tagging corpus for model training.

4. The lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF as described in claim 3, characterized in that: The S2 lightweight DeBERTa encoding layer is composed of multiple stacked DeBERTa encoder modules. Each module contains a decoupled self-attention unit and a feedforward neural network. The lightweight DeBERTa encoding layer uses residual connections and layer normalization for stable training. The decoupled self-attention unit calculates content attention and location attention by separating content features from location information; Content attention With positional attention The calculation formula is as follows: in, For content query matrix; This is the transpose of the content key matrix; This is the transpose of the position key matrix; For the attention dimension; For content attention; For positional attention; Content attention With positional attention The combined attention weight A is obtained by fusion, and its specific expression is as follows: in, To incorporate attention weights; This is the balance coefficient; For content attention; For positional attention; The feedforward neural network sequentially performs linear transformation, nonlinear activation, and Dropout operations; the linear transformation is performed on the input feature matrix. A linear transformation is performed to map the data to a higher-dimensional space. The matrix expression after the linear transformation is as follows: in, for The matrix after linear transformation; For the input feature matrix, The length of the input sequence; The weights of the first-level linear transformation, The dimension of the input feature. The dimension of the intermediate layer of the feedforward neural network; This is the bias vector for the first-level linear transformation; The nonlinear activation uses the ReLU activation function to perform the nonlinear mapping: in, for The activated matrix; for The matrix after linear transformation; For activation functions; The Dropout operation randomly masks some neurons in the nonlinear feature space; After the Dropout operation, the feedforward network performs dimensionality reduction through a second linear transformation to obtain the output features: in, The feature matrix output by the DeBERTa encoding layer. The sequence length of this matrix; for The activated matrix; The weights for the second-level linear transformation. To represent the dimension of the output features of the feedforward network; The dimension of the intermediate layer of the feedforward neural network; This is the bias vector for the second-level linear transformation; The training phase of the lightweight DeBERTa encoding layer is divided into a pre-training phase and a fine-tuning phase. The pre-training phase performs morphological and positional joint masking, masking the prefix and suffix morphological parts of Yi words and recording their positional information; when masking the morphological parts of Yi words, the masking probability formula is: in, Vocabulary in the Yi language part-of-speech tagged corpus; For vocabulary The probability of being masked; For vocabulary The probability of including morphological features is obtained through corpus statistics; The base probability of the random mask; This is the shape weighting coefficient; The pre-training phase implements an embedding sharing mechanism, which is optimized through a joint generator and discriminator. The joint loss function of the feedforward neural network in the pre-training phase is: in, This is the joint loss function of the feedforward neural network; This represents the loss in masked language modeling; This represents the loss for word replacement detection; λ is the balance coefficient. The mask language modeling loss The specific expression is as follows: in, It is the loss for masked language modeling; The expectation for the entire training corpus; For the first in the input sequence Each token location; The set of masked token positions; The original, correct token; This is the masked version of the input sequence; For the generator parameter set; For the generator in the generator parameter set Below, the predicted probability of the masked word; This means summing up all the token positions in the masked token position set C one by one; It is a logarithmic function; The replacement word detection loss The specific expression is as follows: in, It is the loss for replacing words; The expectation for the entire training corpus; For the first in the input sequence Each token location; The input sequence for the discriminator contains real words and replacement words; The first received by the discriminator One token; The original, correct token; The function is an indicator: 1 if the position contains a real word, and 0 otherwise. This determines the probability that the token has been replaced by the discriminator. The parameter set for the discriminator; This means summing up each token position in the input sequence; It is a logarithmic function; The pre-training phase, based on the embedding sharing mechanism, further executes the gradient decoupling embedding sharing mechanism, sharing the word embedding matrix between the generator and discriminator, resulting in the final loss function of the lightweight DeBERTa encoding layer pre-training phase, expressed as: in, The overall goal for optimization throughout the entire pre-training phase; To embed consistency constraints; It is the loss for masked language modeling; It is the loss for replacing words; 1 and 2 is the hyperparameter that balances the weights of the two parts; During the fine-tuning phase, the model employs a bidirectional encoding mechanism, which involves simultaneously performing forward and backward modeling on the input sequence to capture bidirectional contextual dependencies.

5. The lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF as described in claim 4, characterized in that: The compact Transformer encoding layer described in S2 contains multiple simplified Transformer encoding modules. Each simplified Transformer encoding module contains a multi-head self-attention and bottleneck feedforward network. The Transformer encoding layer is trained stably using Dropout and residual connections. The multi-head self-attention calculation The formula for calculating the attention points is: in, For the first The intermediate feature matrix output by each attention head; Indicates the first One point of attention; For the first A matrix of attention head values; Indicates the first Attention weight matrix for each attention head; The first Attention weight matrix for each attention head The expression is as follows: in, Indicates the first Attention weight matrix for each attention head; and The first The query matrix and key matrix of each attention head; Indicates the attention dimension; For activation functions; The first Attention weight matrix for each attention head The expression for sparsification is as follows: in, This is the original attention weight matrix; This is the sparsed attention matrix; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise; is the global attention entropy threshold; i represents the position of the i-th token in the input sequence; j represents the position of the j-th token in the input sequence; The bottleneck feedforward network will output the feature matrix from the DeBERTa encoding layer. The feature dimension is increased by a linear layer, then reduced by activation, and the transformation formula is: in, is the intermediate feature matrix after the first linear transformation and ReLU activation, and n is the sequence length of this matrix; , These are the weights and bias vectors for the first-level linear transformation, respectively. , These are the weights and bias vectors for the second-level linear transformation, respectively; The feature matrix output by the DeBERTa encoding layer; This is the output matrix of the second linear transformation; The bottleneck feedforward network introduces embedding sharing and discriminative optimization, as shown in the following expression: in, For the input feature representation of the embedded sharing mechanism; , These are the weights and bias vectors for the first-level linear transformation, used for dimensionality increase; , These are the weights and bias vectors for the second-layer linear transformation, used for dimensionality reduction. This is the output matrix of the second linear transformation; It is a non-linear activation function; The Transformer encoding layer employs Dropout and residual connections for stable training, and introduces low-dimensional manifold mapping and sparse attention constraint mechanisms. The bottleneck feedforward network uses nonlinear feature mapping. in, This is the feature representation after two-level linear transformation and nonlinear mapping; The feature matrix output by the DeBERTa encoding layer; It is a non-linear activation function; , These are the weights and bias vectors for the first-level linear transformation, used for dimensionality increase; , These are the weights and bias vectors for the second-layer linear transformation, used for dimensionality reduction. The output matrix of the second linear transformation Introducing the manifold regularization term, the formula for the manifold regularization term is: in, For manifold regularization; , For the first The first sample and the first High-dimensional feature representation of each sample; The distance is Euclidean. Similarity weights; , Indicates the first position in the input sequence The and the first Index of each sample; It is a 2-norm; This indicates that the summation is performed on all samples in the input sequence.

6. The lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF as described in claim 1, characterized in that: The optimized CRF decoding layer described in S2 includes a linear projection layer and a conditional random field layer, and the CRF decoding layer is trained and inferred. The linear projection layer maps 256-dimensional features to the label space, generating an emission score matrix, the expression of which is as follows: in, For the emission fraction matrix, The number of samples in a batch. The sequence length of this matrix; The linear projection weight matrix; It is the bias vector; For the input feature representation of the embedded sharing mechanism; The conditional random field layer is trained using a label transition matrix. Model the dependency between labels, assign learnable parameters only to high-frequency transition rules, and assign penalty values ​​to all low-frequency transition paths other than high-frequency transition rules; The loss calculation during the CRF layer training phase, combined with the high-frequency transition rule, is as follows: in, is the loss function of the CRF layer; N is the number of training samples, and c represents the c-th training sample in the training set; The input sequence corresponding to the c-th training sample; To and The corresponding label sequence; For a CRF decoding layer in a given input sequence Under the condition, for the tag sequence The conditional probability; log is the natural logarithm; Indicates to Summing the training samples; in The expression is as follows: in, For a CRF decoding layer in a given input sequence Under the condition, for the tag sequence The conditional probability; For position Corresponding tags Launch fraction; Indicates from the label Transition to tags The score; only when the transfer is to Participate in learning at any time, A set of high-frequency transfer rules; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. Represents the first in the sequence One location; It is an exponential function; Will The total score of the entire sequence is obtained by summing the launch score and transfer score at each position; To traverse all possible label paths Calculate the exponent of its score; Represents all possible combinations of label sequences; express In position Tags; For position Corresponding tags Launch fraction; Indicates from the label Transition to tags The score; The training inference includes a training phase and an inference phase. The training phase requires calculating the joint probability of each label sequence, the specific expression of which is as follows: in, For joint probability; As a normalization factor, the loss function takes the negative log-likelihood; Emission fractions generated for the linear projection layer; The transition score in the label transition matrix represents the transition score from the previous label. Transition to current tag The score; Represents the first in the sequence The location index of each token; To and The corresponding label sequence; Indicates position Tags; To obtain the total score for the complete label sequence by summing the scores at each position; ) is an exponential function; In the inference phase, the Viterbi algorithm is used to search for the optimal label path and obtain the final part-of-speech label for each word. In the parameter optimization of the conditional random field layer, L1 sparse regularization constraints are introduced. By penalizing the sum of the absolute values ​​of the parameters, the conditional probability is defined as: in, For conditional probability; Normalization factor; For the first One characteristic function; For all time steps With characteristic function Summation; These are the weights learned from the linear projection layer and the transition matrix; It is the characteristic function; It is an exponential function; The tag for the previous time step; This is the label for that time step; For time step Input token; And add a sparse regularization term: in, For sparse regularization terms; For sparse constraint terms; For the smoothing constraint term, and These are the hyperparameters of the sparse regularization term; For the first The input sequence of one sample; The label sequence corresponding to the input sequence; It is a logarithmic function; For model parameters The conditional probability under the following conditions; Represents the first in the training set One sample; These are model parameters; It is a norm; It is a 2-norm; To Summation is performed on all samples.

7. The lightweight Yi language part-of-speech tagging method based on DeBERTa+Transformer+CRF as described in claim 1, characterized in that: The knowledge distillation described in S3 uses the baseline model of the DeBERTa encoding layer without pruning or quantization as the teacher model to guide the lightweight model's learning. The lightweight model serves as the student model. The quantization training described in S3 uses INT8 quantization. The dynamic pruning described in S3 removes attention heads and neurons that contribute 30-50% of the data. The total loss of the student model is a weighted sum of the KL divergence loss and the CRF layer loss. in, The total loss for the student model; For KL divergence loss; For CRF layer loss; KL divergence loss in the formula for: in, For KL divergence loss; For student models in position For tags The probability distribution; For the teacher model in position For tags The probability distribution; This represents the total number of tags; Indicates the label in the sequence; It is a logarithmic function; Indicates to Summation of the labels; Indicates the sequence Sum of positions.

Images