Lightweight natural language processing large model training method

Abstract

The invention discloses a lightweight natural language processing large model training method. Comprising the following steps: obtaining processing language data; a labeled enhanced data set is obtained; a sparse mechanism of the sub-network is dynamically activated through semantic complexity, and an activation mask is obtained through calculation; generating a quantization bit width by activating the parameter sensitivity of the mask; performing cross feedback adjustment on the mixing precision quantification strategy; and evaluating the trained student model. According to the method, through dynamic sparse activation, mixing precision quantification and collaborative optimization, the core problems of static strategy stiffness, quantification precision loss, low knowledge migration efficiency and the like in large model lightweight are solved; dynamic sparse activation replaces traditional static pruning, and semantic loss is reduced; in order to realize optimized feature extraction of the enhanced data set, fusion optimization features are adopted; the mixing precision quantification effectively reduces the complexity of calculation.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language models, and more specifically, particularly relates to a lightweight natural language processing large model training method. Background Art

[0002] In recent years, the field of natural language processing (NLP) has made significant progress through large-scale pre-trained models (such as BERT and GPT-3). However, with the exponential growth of model parameters (from billions to trillions), training and deploying these large models face huge computing resource consumption and storage costs. The current mainstream technical solutions have the following key problems in the lightweight process: Traditional model compression methods (such as pruning and parameter freezing) rely on static strategies, which limits the ability to express features. Fixed pruning rate problem: Typical pruning methods achieve compression by removing low-amplitude weights, but the pruned model cannot dynamically restore the removed parameters. Poor task adaptability: The statically pruned model performs significantly worse in complex tasks (such as multi-language translation and long text generation).

[0003] Existing quantization techniques use the same bit width for all parameters, ignoring differences in parameter sensitivity; performance collapse of sensitive layers: key parameters in the attention mechanism (such as the Query-Key projection matrix) are sensitive to quantization errors. Insufficient dynamic range adaptation: Uniform quantization cannot adapt to the heterogeneity of parameter distribution.

[0004] Traditional distillation methods fail when the capabilities of the teacher and student models differ greatly; Capacity mismatch problem: When the difference in the number of parameters between the teacher model and the student model exceeds 100 times, the student model cannot effectively imitate the teacher's behavior, resulting in a 23% increase in the perplexity of text generation after distillation. Information loss bottleneck: Single-stage distillation is difficult to balance knowledge transfer of different granularities.

[0005] Deficiencies of existing technical solutions: How to compress the model size while maintaining adaptability to complex semantic tasks. Existing methods lack dynamic perception mechanisms and cross-module collaborative optimization, resulting in the following problems: static compression strategies cannot respond to dynamic changes in input semantics; coarse-grained quantization causes performance degradation of key modules; and the distillation process produces knowledge residues due to the gap in teacher-student capabilities. Therefore, we propose a lightweight natural language processing large model training method. Summary of the invention

[0006] In view of the problems existing in the prior art, the purpose of the present invention is to provide a lightweight natural language processing large model training method, which adaptively adjusts the parameter participation by input semantic complexity, allocates the appropriate quantization bit width based on parameter sensitivity, and realizes the joint training of sparse activation, hybrid quantization and knowledge distillation, thereby solving the problems of static strategy rigidity, quantization accuracy loss, inefficient knowledge transfer, etc. in the lightweight quantization of large models, and effectively improves the computational efficiency and accuracy of the model.

[0007] To achieve the above object, the present invention provides the following technical solution: a lightweight natural language processing large model training method, comprising the following steps: Collect a large amount of natural language data, pre-process the natural language data, and obtain processed language data; Select a mature large-scale natural language model as the teacher model, and train the processed language data through the teacher model to obtain a labeled enhanced data set; The lightweight natural language processing large model is used as the student model, and the enhanced data set is input into the student model for training. The semantic complexity of the enhanced data set is calculated, and the activation mask is calculated by dynamically activating the sparse mechanism of the sub-network through the semantic complexity. The calculated activation mask is input into a preset mixed precision quantization strategy, parameter sensitivity of the activation mask is calculated, and the quantization bit width is generated by the parameter sensitivity of the activation mask; Then the activation mask and quantization bit width are combined to complete the calculation, realizing cross-feedback adjustment of the mixed precision quantization strategy; The natural language data is input into the student model, and the student model outputs the training results. The trained student model is evaluated based on the training results, and when the evaluation results meet the predetermined conditions, the current student model is output, which is the natural language processing model for natural language processing.

[0008] Further preferably, the acquisition of the processed language data is preprocessing the natural language data, and the preprocessing steps include text cleaning, spelling correction, duplicate data removal, and text cutting and splitting; The text cleaning is used to remove special symbols, line breaks and spaces from natural language data; The spelling correction is used to correct errors in natural language data and to unify the uppercase and lowercase format of text in the natural language data; The deduplication data is used to remove repeated texts or samples with the same annotations in natural language data, which helps to avoid model overfitting; The text cutting and splitting is used to cut or split long texts of natural language data, especially for models with maximum length restrictions, by splitting the long text into multiple paragraphs or sentences, so that each part is easy for the model to process.

[0009] Further preferably, the annotated enhanced data set is obtained by the following method: The teacher model generates pseudo labels: the teacher model is used to predict or annotate the processed language data to generate pseudo label data; Pseudo-label screening: Screen the generated pseudo-label data and filter out pseudo-label data with low confidence by setting the confidence level; Augmented dataset: Combine pseudo-labeled data with processed language data to generate a new augmented dataset.

[0010] Further preferably, the calculation formula for setting the confidence is as follows: The confidence setting adopts a dynamic threshold as the threshold, which is set according to the average confidence of all natural language data output by the teacher model. The formula is as follows: ; in, is the average confidence level, is the amount of processed language data, It is The prediction confidence for processing language data.

[0011] Further preferably, the steps of the sub-network sparseness mechanism are as follows: Extract features from the input annotated augmented data set, fuse the extracted features to generate fused features, and then generate multi-dimensional semantic representations, and use the multi-dimensional semantic representations as the basis for sparse activation decisions; The activated sub-network is dynamically selected based on the input fusion features, and the activation mask is calculated based on the semantic complexity; And the steps for feature extraction for the enhanced data set are as follows: The enhanced data set is segmented, and features are extracted from the segmented enhanced data set. The extracted features are then fused, and features are extracted again from the fused features. First, extract features from each character in the enhanced data set, then fuse each extracted feature and extract features again to optimize features for all characters; And extract features from each word in the enhanced data set, then fuse each extracted feature, extract features again, and optimize features for all words; The enhanced data set is segmented again, segmented evenly by character numbers, and then the enhanced data set is divided into two segments, and then feature extraction is performed on each segment of the enhanced data set and the complete enhanced data set, and then the extracted features are fused and feature extraction is performed again to optimize the context features; Finally, the optimized character features, optimized vocabulary features and optimized context features are fused to obtain fused features.

[0012] Further preferably, the fused features include character-level features, word-level features and context features, and the enhanced dataset uses a lightweight encoder to extract character-level features, word-level features and context features; The calculation formula of the fusion feature is as follows: ; in, Represented as fusion features, Represented as character-level features, Represented as word-level features, Represented as contextual features.

[0013] Further preferably, the semantic complexity evaluation module implements context dimension analysis processing on the enhanced data set through a dual-channel feature extractor, and completes a real-time entropy value calculation engine; The analysis and calculation of the context dimension are as follows: ; in, Represented as the context dimension feature of the enhanced dataset, Represented as the hidden state output by the basic Transformer encoder of the augmented dataset; Represented as the document input of the augmented dataset, Represented as multi-head self-attention features, Represented as a hierarchical LSTM feature, Represented as document topic vector; Dual-channel fusion strategy: Dynamic Gating Fusion Algorithm: ; in, Represented as a multi-dimensional semantic representation, Represented as a learnable weight matrix, Represented as Sigmoid activation function, Represented as a text dimension representation, Represented as context dimension representation; The real-time entropy value calculation engine calculates as follows: Definition and calculation of semantic entropy: Multi-granularity entropy value fusion formula: ; in, Expressed as multi-granularity entropy value, Expressed as Token-level entropy, Expressed as sentence-level entropy, Expressed as document-level entropy, They are respectively expressed as the weight coefficient of token-level entropy, the weight coefficient of sentence-level entropy, and the weight coefficient of document-level entropy; The entropy calculation of each level is: Token-level entropy:

[0014] in, Represented as predefined semantic categories, Represented as the number of Tokens in the enhanced dataset, Represented as a given Token , which belongs to the category The probability of Expressed as Tokens; Sentence-level entropy: ; in, Denoted as the depth distribution of the augmented dataset, Represented as a uniform distribution, Represented as the number of sentences currently being processed, Expressed as KL divergence, measuring the actual distribution With uniform distribution differences; Document-level entropy: ; in, Expressed as the topic distribution probability of the enhanced dataset, Expressed as the number of topics.

[0015] Further preferably, the parameter sensitivity of the activation mask generates the quantization bit width, and the parameter sensitivity of the activation mask is calculated as follows: ; Among them, the activation frequency of the activation mask For the recent The number of times it is activated in the step, =0.9 is the attenuation coefficient, Represents the previous time step The exponentially weighted activation frequency of Represents an indicator function, indicating the parameter At time step Is it activated? Represents the weight coefficient of the current activation state, and complementary; Parameter sensitivity calculation: ; in, Expressed as The parameter sensitivity of the activation mask, and is expressed as a weight coefficient to balance the contribution of activation frequency and parameter sensitivity, and =0.6, =0.4, It is expressed as the normalized activation frequency, Expressed as normalized gradient sensitivity; The quantization bit width is calculated as follows: ; in, Expressed as the value of the quantization bit width, They are represented as the lowest dynamic threshold and the highest dynamic threshold respectively; ,in, Expressed as the 30% quantile of parameter sensitivity, Expressed as the 70% quantile of parameter sensitivity.

[0016] Further preferably, the steps of the cross feedback regulation are as follows: Forward propagation: Generate activation masks based on input, filter activated sub-networks, load quantized weights of activation parameters, and perform calculations; Quantization-aware training: In backpropagation, the pseudo gradient of the quantization parameter is calculated and the activation frequency is updated and gradient sensitivity ; Dynamic adjustment: The sensitivity quantiles are recalculated and the bit width allocation is updated. The activation pattern is counted using a sliding window to optimize the mask generation strategy.

[0017] Further preferably, the student model is evaluated by the following calculation: , in, is the posterior probability, that is, the similarity between the output of the student model and the enhanced data set, N is the total number of training results for comparison and judgment, Represented as the training result, Represented as the position sequence of training results, is the likelihood probability of the similarity between the training results and the enhanced dataset, is the prior probability of the similarity between the training results and the enhanced dataset, is the likelihood probability that the training result is not similar to the enhanced dataset, is the prior probability that the training result is not similar to the enhanced dataset.

[0018] Technical effects and advantages of the present invention: This application solves the core problems of static strategy rigidity, quantization accuracy loss, and inefficient knowledge transfer in lightweight large models through dynamic sparse activation, mixed precision quantization, and collaborative optimization. Dynamic sparse activation extracts features from natural language text, fuses the extracted features and uses them as the decision basis for sparse activation, and calculates the activation mask through the semantic complexity evaluation module. The semantic complexity evaluation module is based on the semantic entropy value calculation engine of the input text, and the semantic complexity evaluation module uses a dual-channel feature extractor to perform characterization analysis of the text dimension and the context dimension, and then performs fusion processing to achieve dynamic routing based on real-time semantic analysis, replacing traditional static pruning and reducing semantic loss; In order to achieve optimized feature extraction of the enhanced data set, fusion optimization features are used, that is, the enhanced data set is subjected to character, vocabulary and context feature extraction, and then the extracted features are fused and processed, and the fused features are extracted again to achieve feature optimization, reduce the complexity of the analysis and processing of the enhanced data set in the natural language processing large model, and improve the efficiency of natural language analysis and processing; Mixed precision quantization calculates parameter gradient sensitivity by activating masks and obtains quantization bit width. That is, parameter gradient sensitivity is calculated by activating masks and the parameters are divided into three levels: critical, medium, and non-critical. This effectively reduces the complexity of the calculation and supports a quantization framework with dynamic bit width switching to improve the accuracy of the model.

[0019] Through dynamic sparsity, hybrid quantization and two-way cross-feedback adjustment, and through joint optimization processing, the optimal balance between model efficiency and accuracy is achieved, solving core problems such as static strategy rigidity, quantization accuracy loss, and inefficient knowledge transfer in the lightweighting of large models.

[0020] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the step flow provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, the lightweight natural language processing large model training method provided by the embodiment of the present invention includes the following steps: Collect a large amount of natural language data, pre-process the natural language data, and obtain processed language data; Select a mature large-scale natural language model as the teacher model, and train the processed language data through the teacher model to obtain a labeled enhanced data set; The lightweight natural language processing large model is used as the student model, and the enhanced data set is input into the student model for training. The semantic complexity of the enhanced data set is calculated, and the activation mask is calculated by dynamically activating the sparse mechanism of the sub-network through the semantic complexity. The calculated activation mask is input into a preset mixed precision quantization strategy, the parameter sensitivity of the activation mask is calculated, and the quantization bit width is generated by the parameter sensitivity of the activation mask; Then the activation mask and quantization bit width are combined to complete the calculation, realizing cross-feedback adjustment of the mixed precision quantization strategy; The natural language data is input into the student model, and the student model outputs the training results. The trained student model is evaluated based on the training results, and when the evaluation results meet the predetermined conditions, the current student model is output, which is the natural language processing model for natural language processing.

[0024] In this embodiment, preferably, the acquisition of the processed language data is to preprocess the natural language data, and the preprocessing steps include text cleaning, spelling correction, duplicate data removal, and text cutting and splitting; The text cleaning is used to remove special symbols, line breaks and spaces from natural language data; The spelling correction is used to correct errors in natural language data and to unify the uppercase and lowercase format of text in the natural language data; The deduplication data is used to remove repeated texts or samples with the same annotations in natural language data, which helps to avoid model overfitting; The text segmentation and splitting is used to segment or split long texts of natural language data, especially for models with a maximum length limit, by splitting the long text into multiple paragraphs or sentences, so that each part is easy for the model to process; It should be noted that by preprocessing natural language data, the format uniformity of natural language can be improved, making it easier to input natural language into the natural language processing model for analysis and processing, and the preprocessed natural language can improve the processing efficiency of the natural language model, reduce errors and improve accuracy.

[0025] In this embodiment, preferably, the annotated enhanced data set is obtained by the following method: The teacher model generates pseudo labels: the teacher model is used to predict or annotate the processed language data to generate pseudo label data; Pseudo-label screening: Screen the generated pseudo-label data and filter out low-confidence pseudo-label data by setting the confidence level; Enhanced dataset: Combine pseudo-labeled data with processed language data to generate a new enhanced dataset; It should be noted that, by processing language data through the teacher model to obtain an enhanced data set, and by training the enhanced data set in the student model, the student model can obtain better generalization ability and performance than its own training by imitating the teacher model, especially in an environment with small sample data or limited resources, and by setting the confidence to filter samples with low confidence, the accuracy of the enhanced data set can be effectively improved, which is convenient for the student model to be quickly trained and used.

[0026] In this embodiment, preferably, the calculation formula for setting the confidence is as follows: The confidence setting adopts a dynamic threshold as the threshold, which is set according to the average confidence of all natural language data output by the teacher model. The formula is as follows: ; in, is the average confidence level, is the amount of processed language data, It is prediction confidence for processing language data; It should be noted that the confidence uses a dynamic threshold to set the threshold, which can avoid transmitting unreliable information, concentrate on training the student model on the more reliable predictions of the teacher model, and improve the efficiency of distillation. By dynamically adjusting the intensity of information transmission, the student model can better adapt to different types of samples, thereby improving its generalization ability. The model can automatically adjust the distillation process according to different data distributions and adapt to more complex training scenarios. The confidence of the teacher model uses a dynamic threshold as the threshold, which is a technology that can enhance the knowledge distillation process. By dynamically adjusting the threshold according to the characteristics of different input samples or the prediction confidence of the teacher model, the student model can more selectively learn the knowledge of the teacher model and improve the effect of distillation.

[0027] In this embodiment, preferably, the steps of the sub-network sparseness mechanism are as follows: Extract features from the input annotated augmented data set, fuse the extracted features to generate fused features, and then generate multi-dimensional semantic representations, and use the multi-dimensional semantic representations as the basis for sparse activation decisions; The activated sub-network is dynamically selected based on the input fusion features, and the activation mask is calculated based on the semantic complexity; And the steps for feature extraction for the enhanced data set are as follows: The enhanced data set is segmented, and features are extracted from the segmented enhanced data set. The extracted features are then fused, and features are extracted again from the fused features. First, extract features from each character in the enhanced data set, then fuse each extracted feature and extract features again to optimize features for all characters; And extract features from each word in the enhanced data set, then fuse each extracted feature, extract features again, and optimize features for all words; The enhanced data set is segmented again, segmented evenly by character numbers, and then the enhanced data set is divided into two segments, and then feature extraction is performed on each segment of the enhanced data set and the complete enhanced data set, and then the extracted features are fused and feature extraction is performed again to optimize the context features; Finally, the optimized character features, optimized vocabulary features and optimized context features are fused to obtain fused features; It should be noted that the computational efficiency and performance of the network are improved through the sparse activation mechanism. Through the strategies of feature extraction, feature fusion, multi-dimensional semantic representation generation and dynamic selection of sub-network modules, the model can intelligently select which modules are most important under given input, thereby reducing unnecessary calculations. With the support of the semantic complexity evaluation module, the model can more accurately control the activated sub-networks and improve the efficiency of training and reasoning, especially when processing complex or diverse data.

[0028] In this embodiment, preferably, the fused features include character-level features, word-level features and context features, and the enhanced data set uses a lightweight encoder to extract character-level features, word-level features and context features; The calculation formula of the fusion feature is as follows: ; in, Represented as fusion features, Represented as character-level features, Represented as word-level features, Represented as contextual features; It should be noted that by analyzing the character-level features, word-level features, and contextual features of the enhanced dataset, we can improve our fine-grained understanding of the enhanced dataset, and achieve feature fusion of the enhanced household set to obtain feature information of the enhanced dataset, which is convenient for subsequent calculation and processing.

[0029] In this embodiment, preferably, the semantic complexity evaluation module implements context dimension analysis processing on the enhanced data set through a dual-channel feature extractor, and completes a real-time entropy value calculation engine; The analysis and calculation of the context dimension are as follows: ; in, Represented as the context dimension feature of the enhanced dataset, Represented as the hidden state output by the basic Transformer encoder of the augmented dataset; Represented as the document input of the augmented dataset, Represented as multi-head self-attention features, Represented as a hierarchical LSTM feature, Represented as document topic vector; Dual-channel fusion strategy: Dynamic Gating Fusion Algorithm: ; in, Represented as a multi-dimensional semantic representation, Represented as a learnable weight matrix, Represented as Sigmoid activation function, Represented as a text dimension representation, Represented as context dimension representation; The real-time entropy value calculation engine calculates as follows: Definition and calculation of semantic entropy: Multi-granularity entropy value fusion formula: ; in, Represented as activation mask, Expressed as Token-level entropy, Expressed as sentence-level entropy, Expressed as document-level entropy, They are respectively expressed as the weight coefficient of token-level entropy, the weight coefficient of sentence-level entropy, and the weight coefficient of document-level entropy; The entropy calculation of each level is: Token-level entropy:

[0030] in, Represented as predefined semantic categories, Represented as the number of Tokens in the enhanced dataset, Represented as a given Token , which belongs to the category The probability of Expressed as Tokens; Sentence-level entropy: ; in, Denoted as the depth distribution of the augmented dataset, is represented as a uniform distribution, is represented as the number of sentences currently being processed, Expressed as KL divergence, measuring the actual distribution With uniform distribution differences; Document-level entropy: ; in, It is expressed as the topic distribution probability of the enhanced dataset, It is expressed as the number of topics; Dynamic threshold control strategy: ; in, =0.9 (momentum coefficient), EMA is the exponential moving average, is the entropy value sequence of the last 10 steps, Expressed as a dynamic threshold; It should be noted that the dual-channel fusion strategy is used to achieve feature fusion processing of the upper and lower dimensions of the text and the context, and a multi-dimensional semantic representation is obtained. The dual-channel fusion strategy is used to achieve fusion processing of the multi-dimensional semantic representation and multi-level entropy, and the activation mask is calculated to effectively implement pruning of the enhanced data set and reduce the loss of semantic information.

[0031] In this embodiment, preferably, the parameter sensitivity of the activation mask generates the quantization bit width, and the parameter sensitivity of the activation mask is calculated as follows: ; Among them, the activation frequency of the activation mask For the recent The number of times it is activated in the step, =0.9 is the attenuation coefficient, Represents the previous time step The exponentially weighted activation frequency of Represents an indicator function, indicating the parameter At time step Is it activated? Represents the weight coefficient of the current activation state, and complementary; Parameter sensitivity calculation: ; in, Expressed as The parameter sensitivity of the activation mask, and is expressed as a weight coefficient to balance the contribution of activation frequency and parameter sensitivity, and =0.6, =0.4, It is expressed as the normalized activation frequency, Expressed as normalized gradient sensitivity; The quantization bit width is calculated as follows: ; in, Expressed as the value of the quantization bit width, They are represented as the lowest dynamic threshold and the highest dynamic threshold respectively; ,in, Expressed as the 30% quantile of parameter sensitivity, Expressed as the 70% quantile of parameter sensitivity; It should be noted that by calculating the parameter sensitivity of the activation mask, calculating the parameter sensitivity through gradient sensitivity, and calculating the quantization bit width through parameter sensitivity, and setting a dynamic threshold, the key parameter accuracy of hybrid quantization can be retained, the uniform quantization performance degradation can be avoided, and the proportion of video memory can be reduced.

[0032] In this embodiment, preferably, the steps of the cross feedback regulation are as follows: Forward propagation: Generate activation masks based on input, filter activated sub-networks, load quantized weights of activation parameters, and perform calculations; Quantization-aware training: In backpropagation, the pseudo gradient of the quantization parameter is calculated and the activation frequency is updated and gradient sensitivity ; Dynamic adjustment: Recalculate the sensitivity quantiles and update the bit width allocation; use the sliding window to count the activation patterns and optimize the mask generation strategy; It should be noted that the optimal balance between model efficiency and accuracy is achieved through dynamic sensitivity scoring, mask-aware quantization allocation and bidirectional feedback adjustment.

[0033] In this embodiment, preferably, the student model is evaluated by the following calculation: , in, is the posterior probability, that is, the similarity between the output of the student model and the enhanced data set, N is the total number of training results for comparison and judgment, Represented as the training result, Represented as the position sequence of training results, is the likelihood probability of the similarity between the training results and the enhanced dataset, is the prior probability of the similarity between the training results and the enhanced dataset, is the likelihood probability that the training result is not similar to the enhanced dataset, is the prior probability that the training result is not similar to the enhanced dataset; It should be noted that through the computational analysis of the evaluation, it is possible to match the output of the student model with the results of the teacher model, so as to obtain the accuracy of the student model. When it is determined that the matching probability between the student model and the teacher model reaches 80%, the student model is judged to be usable, and the output of the student model is used.

[0034] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight natural language processing large model training method, characterized in that: The steps include: Collect a large amount of natural language data, pre-process the natural language data, and obtain processed language data; Select a mature large-scale natural language model as the teacher model, and train the processed language data through the teacher model to obtain a labeled enhanced data set; The lightweight natural language processing large model is used as the student model, and the enhanced data set is input into the student model for training. The semantic complexity of the enhanced data set is calculated, and the activation mask is calculated by dynamically activating the sparse mechanism of the sub-network through the semantic complexity. The calculated activation mask is input into a preset mixed precision quantization strategy, parameter sensitivity of the activation mask is calculated, and the quantization bit width is generated by the parameter sensitivity of the activation mask; Then the activation mask and quantization bit width are combined to complete the calculation, realizing cross-feedback adjustment of the mixed precision quantization strategy; The natural language data is input into the student model, and the student model outputs the training results. The trained student model is evaluated based on the training results, and when the evaluation results meet the predetermined conditions, the current student model is output, which is the natural language processing model for natural language processing.

2. The lightweight natural language processing large model training method according to claim 1 is characterized by: The acquisition of the processed language data is to pre-process the natural language data, and the pre-processing steps include text cleaning, spelling correction, removal of duplicate data, and text cutting and splitting; The text cleaning is used to remove special symbols, line breaks and spaces from natural language data; The spelling correction is used to correct errors in natural language data and to unify the uppercase and lowercase format of text in the natural language data; The deduplication data is used to remove repeated texts or samples with the same annotations in natural language data, which helps to avoid model overfitting; The text cutting and splitting is used to cut or split long texts of natural language data, especially for models with maximum length restrictions, by splitting the long text into multiple paragraphs or sentences, so that each part is easy for the model to process.

3. The lightweight natural language processing large model training method according to claim 1 is characterized in that: The annotated enhanced dataset is obtained by the following method: The teacher model generates pseudo labels: the teacher model is used to predict or annotate the processed language data to generate pseudo label data; Pseudo-label screening: Screen the generated pseudo-label data and filter out low-confidence pseudo-label data by setting the confidence level; Augmented dataset: Combine pseudo-labeled data with processed language data to generate a new augmented dataset.

4. The lightweight natural language processing large model training method according to claim 3 is characterized by: The calculation formula for setting the confidence is as follows: The confidence setting adopts a dynamic threshold as the threshold, which is set according to the average confidence of all natural language data output by the teacher model. The formula is as follows: ； in, is the average confidence level, is the amount of processed language data, It is Prediction confidence for processing language data.

5. The lightweight natural language processing large model training method according to claim 1 is characterized in that: The steps of the sub-network sparsification mechanism are as follows: Extract features from the input annotated augmented data set, fuse the extracted features to generate fused features, and then generate multi-dimensional semantic representations, and use the multi-dimensional semantic representations as the basis for sparse activation decisions; The activated sub-network is dynamically selected based on the input fusion features, and the activation mask is calculated based on the semantic complexity; And the steps for feature extraction for the enhanced data set are as follows: The enhanced data set is segmented, and features are extracted from the segmented enhanced data set. The extracted features are then fused, and features are extracted again from the fused features. First, extract features from each character in the enhanced data set, then fuse each extracted feature, and extract features again to optimize features for all characters; And extract features from each word in the enhanced data set, then fuse each extracted feature, extract features again, and optimize features for all words; The enhanced data set is segmented again, segmented evenly by character numbers, and then the enhanced data set is divided into two segments, and then feature extraction is performed on each segment of the enhanced data set and the complete enhanced data set, and then the extracted features are fused and feature extraction is performed again to optimize the context features; Finally, the optimized character features, optimized vocabulary features and optimized context features are fused to obtain fused features.

6. The lightweight natural language processing large model training method according to claim 5 is characterized by: The fused features include character-level features, word-level features and context features, and the enhanced dataset uses a lightweight encoder to extract character-level features, word-level features and context features; The calculation formula of the fusion feature is as follows: ； in, Represented as fusion features, Represented as character-level features, Represented as word-level features, Represented as contextual features.

7. The lightweight natural language processing large model training method according to claim 6 is characterized by: The semantic complexity evaluation module implements context dimension analysis processing on the enhanced data set through a dual-channel feature extractor and completes a real-time entropy value calculation engine; The analysis and calculation of the context dimension are as follows: ； in, Represented as the context dimension feature of the enhanced dataset, Represented as the hidden state output by the basic Transformer encoder of the augmented dataset; Represented as the document input of the augmented dataset, Represented as multi-head self-attention features, Represented as a hierarchical LSTM feature, Represented as document topic vector; Dual-channel fusion strategy: Dynamic Gating Fusion Algorithm: ； in, Represented as a multi-dimensional semantic representation, Represented as a learnable weight matrix, Represented as Sigmoid activation function, Represented as a text dimension representation, Represented as context dimension representation; The real-time entropy value calculation engine calculates as follows: Definition and calculation of semantic entropy: Multi-granularity entropy value fusion formula: ； in, Expressed as multi-granularity entropy value, Expressed as Token-level entropy, Expressed as sentence-level entropy, Expressed as document-level entropy, They are respectively expressed as the weight coefficient of token-level entropy, the weight coefficient of sentence-level entropy, and the weight coefficient of document-level entropy; The entropy calculation of each level is: Token-level entropy: ； in, Represented as predefined semantic categories, Represented as the number of Tokens in the enhanced dataset, Represented as a given Token , which belongs to the category The probability of Expressed as Tokens; Sentence-level entropy: ； in, Denoted as the depth distribution of the augmented dataset, Represented as a uniform distribution, Represented as the number of sentences currently being processed, Expressed as KL divergence, measuring the actual distribution With uniform distribution differences; Document-level entropy: ； in, It is expressed as the topic distribution probability of the enhanced dataset, Expressed as the number of topics.

8. The lightweight natural language processing large model training method according to claim 7 is characterized in that: The parameter sensitivity of the activation mask generates the quantization bit width, and the parameter sensitivity of the activation mask is calculated as follows: ； Among them, the activation frequency of the activation mask For the recent The number of times it is activated in the step, =0.9 is the attenuation coefficient, Represents the previous time step The exponentially weighted activation frequency of Represents an indicator function, indicating the parameter At time step Is it activated? Represents the weight coefficient of the current activation state, and complementary; Parameter sensitivity calculation: ； in, Expressed as The parameter sensitivity of the activation mask, and is expressed as a weight coefficient to balance the contribution of activation frequency and parameter sensitivity, and =0.6, =0.4, It is expressed as the normalized activation frequency, Expressed as normalized gradient sensitivity; The quantization bit width is calculated as follows: ； in, Expressed as the value of the quantization bit width, They are represented as the lowest dynamic threshold and the highest dynamic threshold respectively; ,in, Expressed as the 30% quantile of parameter sensitivity, Expressed as the 70% quantile of parameter sensitivity.

9. The lightweight natural language processing large model training method according to claim 8 is characterized by: The steps of the cross-feedback regulation are as follows: Forward propagation: Generate activation masks based on input, filter activated sub-networks, load quantized weights of activation parameters, and perform calculations; Quantization-aware training: In backpropagation, the pseudo gradient of the quantization parameter is calculated and the activation frequency is updated and gradient sensitivity ; Dynamic adjustment: Step 1: Recalculate the sensitivity quantile and update the bit width allocation; Use sliding windows to count activation patterns and optimize mask generation strategies.

10. The lightweight natural language processing large model training method according to claim 1 is characterized by: The student model is evaluated by the following calculation: ， in, is the posterior probability, that is, the similarity between the output of the student model and the enhanced data set, N is the total number of training results for comparison and judgment, Represented as the training result, Represented as the position sequence of training results, is the likelihood probability of the similarity between the training results and the enhanced dataset, is the prior probability of the similarity between the training results and the enhanced dataset, is the likelihood probability that the training result is not similar to the enhanced dataset, is the prior probability that the training result is not similar to the enhanced dataset.

Images