A natural language understanding model training method and system based on width learning

Abstract

This invention discloses a training method and system for a natural language understanding model based on breadth learning, comprising the following steps: preprocessing training data; fusing a pre-trained language model based on the Transformer architecture with breadth learning to obtain a fused model; using the training data as input to the pre-trained language model based on the Transformer architecture, and utilizing the ridge regression learning mechanism of BLS to obtain the corresponding BLS learning evaluation results according to the labels of the adapted natural language understanding task; formulating an appropriate adaptive layer freezing strategy to determine the pre-frozen layer based on the adaptation results; determining the final frozen layer based on the pre-frozen layer, the complexity of the adapted task, and the current computing resources, through an efficient fine-tuning strategy for layer freezing parameters; adding an efficient parameter fine-tuning module to the unfrozen layer to achieve freeze fine-tuning with the introduced additional trainable parameters, thereby obtaining the vertical domain model for the corresponding downstream task of the fine-tuned data.

Description

Technical Field

[0001] This invention relates to the field of automotive electric drive system components technology, and in particular to a method and system for training a natural language understanding model based on width learning. Background Technology

[0002] Currently, efficient parameter fine-tuning methods mainly achieve downstream task adaptation by freezing the parameters of the original pre-trained language model and fine-tuning the additional adapters or low-rank heavy parameterized matrices (LoRA) introduced at each layer. Adapter-based efficient fine-tuning methods reduce the introduction of additional training parameters by introducing tensor product operations to decompose the upsampling and downsampling weight matrices of the adapter into two parts: shared across all layers and customized layer by layer. However, sharing parameters across all layers severely limits the expressive power of the pre-trained language model, weakening its generalization ability on some tasks and prolonging its convergence time. Low-rank heavy parameterized matrix-based efficient fine-tuning methods utilize singular value decomposition to reparameterize the weight matrix into the product of two orthogonal matrices and eigenvalues ​​forming a diagonal matrix. They then use parameter sensitivity to dynamically prune the eigenvalues ​​of the diagonal matrix to adjust the rank of the weight matrix, thereby dynamically allocating different trainable parameters to the weight matrices of different layers based on their adaptation characteristics to optimize model performance. However, this approach is extremely complex to implement, and different hyperparameter settings significantly affect the pruning results, leading to unstable model performance and making it difficult to apply to practical fine-tuning scenarios.

[0003] On the other hand, existing layer-freezing fine-tuning methods mainly employ two layer selection strategies: adaptive and heuristic. Heuristic layer selection strategies design a stepwise layer-freezing strategy by observing the convergence speed trend of the model after the fact, or by exhaustively exploring different layer combinations and selecting the optimal layer combination based on experimental results. However, these methods cannot generalize to multiple models, lack versatility, and consume a lot of resources. Adaptive layer selection strategies design a predictor using an attention mechanism to adaptively select the layers that need fine-tuning for different models. However, training the predictor requires knowing a fully fine-tuned and converged model, which is almost impossible for large pre-trained language models. Most existing parameter-efficient fine-tuning techniques ignore the differences in fine-tuning between different layer modules. At the same time, most existing layer-freezing techniques either exhaustively explore different layer training combinations and select the optimal layer scheme based on experimental results, or use an attention mechanism to dynamically select fine-tuning layers based on a fully fine-tuned pre-trained model as a reference, or pre-select layers for fine-tuning by calculating the Fisher information matrix index of all parameters. Furthermore, in existing technologies, such as the layer-freezing-based hybrid expert model training system (CN119647560A), although it proposes to use the rate of change of expert selection paths as a freezing index and combine it with a resource allocation mechanism, its evaluation computational overhead is large, the freezing granularity is too coarse, and it needs to deal with a large number of expert path statistics and uneven distribution problems on large pre-trained models, resulting in significant consumption of system communication and storage resources. While the layer-freezing and data filtering method for sparse training (CN119768804A) introduces sequential freezing and data filtering mechanisms to reduce the overall computational resource overhead, its freezing strategy relies too much on heuristic rules and freezes step by step from shallow to deep layers, lacking awareness of the actual model state. On high-complexity models such as large pre-trained language models based on Transformer, its generalization performance is poor, resulting in a serious loss of model performance during the freezing process. Summary of the Invention

[0004] To address the infeasibility or excessive training resource consumption of using full parameter fine-tuning methods for large pre-trained language models to adapt to natural language understanding tasks, this invention primarily utilizes Broad Learning System (BLS) and leverages the low resource consumption advantages of layer freezing and efficient parameter fine-tuning methods to propose an adaptive layer freezing and efficient parameter fine-tuning training method adapted to large pre-trained language models. Considering the layer redundancy in existing efficient parameter fine-tuning methods and the inability of existing layer freezing fine-tuning methods to directly adapt to large pre-trained language models, or the high training resource consumption of non-adaptive layer selection methods, BLS, as a general simplified neural network architecture, accurately evaluates the relevance of each layer to the adapted natural language understanding task. Based on the evaluation results, an adaptive layer selection strategy is designed, and layer redundancy is resolved by combining adapter or low-rank adaptation modules for fine-tuning. Given that width learning can quickly and accurately predict the correlation between input features and the labels of the adapted task using only a small number of training samples via ridge regression, it can greatly reduce the training cost of the adaptive layer selection stage. At the same time, by selecting only a portion of the layers and combining them with the adapter for fine-tuning, and freezing the original model parameters, it can greatly reduce the number of parameters and storage usage for fine-tuning large pre-trained language models, thereby further achieving efficient fine-tuning.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] The training method for natural language understanding models based on width learning includes the following steps:

[0007] A fusion model is obtained by fusing a pre-trained language model based on the Transformer architecture with width learning;

[0008] The training data is used as input to a pre-trained language model based on the Transformer architecture. The ridge regression learning mechanism of BLS is used to obtain the evaluation results based on the labels of the adapted natural language understanding task.

[0009] Based on the adaptation results, formulate an appropriate adaptive layer freezing strategy to determine the pre-frozen layer;

[0010] Based on the pre-frozen layer, the complexity of the adaptation task, and the current computing resources, the final frozen layer is determined through an efficient fine-tuning strategy for layer freezing parameters. An efficient fine-tuning module is added to the unfrozen layer to achieve freeze fine-tuning with the introduced additional trainable parameters, thereby obtaining the vertical domain model of the downstream task corresponding to the fine-tuned data.

[0011] Furthermore, the integration method is as follows:

[0012] Suppose the pre-trained language model consists of n stacked Transformer layers. The output of each Transformer layer serves as the input to each layer's BLS. Training data is input into each Transformer layer, first passing through an Attention mechanism to obtain attention representations between different tokens. The multi-head attention results are then concatenated and fed into a feedforward neural network FFN1 to obtain representations between different attentions. After residual and normalization (Add & Norm) operations, the data is fed into another feedforward neural network FFN2 to obtain deep feature representations. Another residual and normalization operation is performed to obtain the output of the current Transformer layer. This output is used as the input to BLS, combined with the target ground truth label, and BLS is used to obtain the relevance of the transformed features of the current Transformer layer to the adapted natural language understanding task.

[0013] Furthermore, obtaining the evaluation results includes the following steps:

[0014] First, assume that the feature of the input text X after being mapped by the word embedding layer is X′, that is...

[0015] X′=XW p-emb (1)

[0016] Among them, W p-emb This refers to the mapping weights of the Embedding layer. Using the word embedding-mapped features X′ as input to the entire fusion model, the output features h of each Transformer FFN2 network layer are as follows:

[0017] h=Norm((Norm(X′+Attn(X′)W o (2)

[0018] Where Norm(·) represents the normalization operation, W o W1 and W2 are known after the pre-trained language model. Attn(·) represents the attention calculation operation, which is calculated as follows:

[0019]

[0020] Where Q = X′W Q K = X′W K V=X′W V ,d k W represents the dimension of vector K. Q W k and W V In a pre-trained language model, the parameters are known.

[0021] After obtaining the output features h∈R of the FFN2 network for each Transformer layer... d Then, using this as the input feature of BLS, we first construct the mapping feature Z, as shown below:

[0022] Z=φ(hW e +β e (4)

[0023] Among them W e ∈R d×r and β e ∈R r It is obtained by random sampling from a standard normal distribution, and φ(·) performs the normalization operation. The specific calculation is as follows:

[0024]

[0025] Where d is the hidden layer dimension of the entire enhancement layer features and the mapped features, r represents the rank, and r << d, z i That is, the i-th feature element of the mapping feature Z;

[0026] The expressive power of the mapping features is optimized by reconstructing them and adding L1 norm for sparsity constraints. The specific reconstruction and optimization objectives are as follows:

[0027]

[0028] The final mapping layer features are represented as follows:

[0029]

[0030] in The mapping weights represent the parameters of the sparse autoencoder, used to map X′ to a mapping feature Z with sparse optimization. s After obtaining the sparsely optimized mapping layer features Z s Then, the enhancement layer node H is constructed using the following calculation method:

[0031] H= ξ(Z s W h +β h (8)

[0032] Among them W h ∈R r×d-r and β h ∈R d-r ξ(·) is a nonlinear activation function obtained by random sampling from a standard normal distribution;

[0033] The enhancement layer feature H and the mapping layer feature Z are combined. sThe hidden layer vectors are obtained by concatenation. The mapping relationship W between the hidden layer features and the label Y of the adapted natural language understanding task is learned. Specifically, the learning is achieved by solving the following optimization objective:

[0034]

[0035] Where λ is a hyperparameter that is manually set, representing the penalty term. Limitations of results;

[0036] Solving using ridge regression, the final mapping relationship W can be represented as follows:

[0037] W=(λI+AA T ) -1 A T Y (11)

[0038] Where I is an identity matrix, and A = [Z] s |H];

[0039] Finally, the fit between the FFN2 output features of each BLS layer and the final adaptation task label, or the evaluation result, is expressed as follows:

[0040] Y eval =AW (12).

[0041] Furthermore, developing an appropriate adaptive layer freezing strategy includes the following steps:

[0042] S41. Assume the pre-trained language model is composed of n layers of Transformers stacked together. Let the evaluation result of each BLS layer be... The pre-frozen layer sequence is obtained by sorting the evaluation results of different layers.

[0043] S42. Let S be the set of pre-frozen layer sequences, initialized as the set of all layer numbers sorted in order, S = {1, 2, 3, ..., n};

[0044] S43. Based on the evaluation results of each layer Rearrange the elements in set S in ascending order to obtain the initial pre-frozen layer sequence.

[0045] Furthermore, determining the final frozen layer through an efficient fine-tuning strategy for layer freezing parameters includes the following steps:

[0046] Based on the given natural language understanding task, determine the final number of layers k to freeze:

[0047] k = floor(len(S) × α) (13)

[0048] Where floor(·) represents floor operation, len(·) represents the number of elements in the set to be calculated, S represents the initial pre-frozen layer sequence, and α is a hyperparameter used to achieve a tradeoff between model performance and training efficiency; the layers corresponding to the first k elements of the initial pre-frozen layer sequence S, i.e. the k layers with the lowest importance, are directly frozen.

[0049] Furthermore, the parameter efficient fine-tuning module (PEFT) is a low-rank heavy parameterized matrix LoRA. The input training data first passes through the Attention mechanism of the Transformer architecture to obtain the corresponding output. If LoRA is inserted next to the Attention mechanism, the input training data will also pass through LoRA to obtain the corresponding output, and be merged with the output of Attention as the input of the feedforward neural network FFN1. Then, it passes through the feedforward neural network FFN1 to obtain the corresponding output, and the output is used as the input of the feedforward neural network FFN2 to obtain the corresponding output. Similarly, if LoRA is selected to be inserted, the corresponding input will also be sent to LoRA and merged with the original FFN2 output.

[0050] Furthermore, the Parameter Efficient Fine-Tuning (PEFT) module is an adapter. The input training data first passes through the Attention mechanism of the Transformer architecture to obtain the corresponding output, and then passes through the feedforward neural network FFN1 to obtain the corresponding output. An adapter is then inserted after the FFN1 network. Subsequently, the output of FFN1 is sent to the adapter for processing to obtain the corresponding output, which is then processed by the residual + normalization (Add & Norm) operation. After obtaining the processed output, if the adapter is selected to be inserted, the output of the original FFN2 is processed by the adapter again to obtain the corresponding output. Finally, a residual + normalization operation is performed again to obtain the final output.

[0051] A system for implementing the aforementioned method for training a natural language understanding model based on width learning includes:

[0052] The data preprocessing module preprocesses the training data.

[0053] The model architecture fusion module is designed with a brand-new width learning and Transformer joint learning module to prepare for the implementation of the adaptive layer freezing module;

[0054] By integrating the selected pre-trained language model architecture type with the corresponding width learning design, a joint learning architecture for the pre-trained language model is obtained.

[0055] The adaptive layer freezing module, based on the data preprocessing module and the model architecture fusion module, obtains the corresponding pre-frozen layer sequence through a forward computation process using training data.

[0056] The efficient fine-tuning module for layer freezing parameters, based on the pre-frozen layer sequence obtained by the adaptive layer freezing module, achieves the final efficient fine-tuning process for layer freezing based on natural language understanding tasks through an efficient fine-tuning strategy for layer freezing parameters.

[0057] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0058] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0059] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0060] 1. This invention proposes a fusion method of a pre-trained language model based on the Transformer architecture and a wide learning architecture, providing a reference and further optimization possibilities for exploring joint learning of wide learning and pre-trained language models based on the Transformer architecture.

[0061] 2. This invention proposes an efficient adaptive layer selection strategy based on width learning. This strategy overcomes the difficulties of existing layer freezing methods, which cannot be directly applied to large pre-trained language models or require a large amount of training resources. By leveraging the advantages of width learning as a general simplified neural network architecture, this layer selection strategy can satisfy modality independence, can be generalized to different modal models based on the Transformer architecture, and can quickly obtain the layers to be frozen using only a small number of training samples through a single forward computation process, greatly reducing the consumption of computational resources.

[0062] 3. This invention combines the adaptive layer freezing method with the efficient parameter fine-tuning method, which solves the layer redundancy problem of the existing efficient parameter fine-tuning method and further reduces the number of training parameters required when the model is adapted to the natural language understanding task, thereby optimizing the storage occupation of model training; at the same time, by freezing the layers that do not need to be fine-tuned, the perturbation introduced by unnecessary layer information is avoided, which can improve the generalization performance of the model on the natural language understanding task. Attached Figure Description

[0063] Figure 1 This is a flowchart of the natural language understanding model training method based on width learning according to an embodiment of the present invention;

[0064] Figure 2 This is a flowchart illustrating the model architecture fusion in an embodiment of the present invention;

[0065] Figure 3 This is a flowchart illustrating the integration of Transformer and BLS in an embodiment of the present invention;

[0066] Figure 4 This is a flowchart illustrating the efficient fine-tuning of layer freezing parameters in an embodiment of the present invention;

[0067] Figure 5 This is a flowchart illustrating the combination of the Transformer architecture and the PEFT module in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer and more explicit, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0069] like Figure 1 As shown in this embodiment, a natural language understanding model training method based on width learning includes the following steps:

[0070] S1. Preprocess the input data of the training samples to unify the input and output formats.

[0071] In order to be applicable to different natural language understanding tasks, the training sample input data first needs to be preprocessed to unify the input and output formats.

[0072] To facilitate joint learning between pre-trained language models (natural language understanding models) based on the Transformer architecture and breadth learning, and to easily handle multiple natural language understanding tasks, the input text is first concatenated with instructions representing the task format as follows:

[0073] {Enter text} => {Command / Instruction | Enter text}

[0074] The concatenated text is used as input text, and the text labels are simultaneously converted into their corresponding numeric forms. For example, if the text labels correspond to {'A','B','C','D'}, they are converted into the corresponding numeric forms {0,1,2,3}, thus unifying the input and output formats of width learning and large pre-trained language models.

[0075] S2. Design a joint learning architecture for width learning and pre-trained language model based on the selected pre-trained language model architecture type and width learning mode.

[0076] As one embodiment, the pre-trained language model architecture in this embodiment is a Transformer-based pre-trained language model, which is divided into three categories: Encoder-Only, Decoder-Only, and Encoder-Decoder. In order to enable joint learning between width learning and the pre-trained language model, different pre-trained language models and corresponding width learning methods are fused to obtain a fused model.

[0077] like Figure 2 As shown, for pre-trained language models with Encoder-Only or Decoder-Only architectures, this embodiment directly uses the output features of the final feedforward neural network (FFN) inside each Transformer layer as the input for width learning; for pre-trained language models with Encoder-Decoder architectures, since the learning and prediction method of this architecture is consistent with the Decdeer-Only architecture, the output features of the FFN network inside each Transformer layer of the Decoder part of this architecture are used as the input for the corresponding width learning.

[0078] A pre-trained language model is formed by stacking n Transformers, where each Transformer layer takes the output of the previous layer as its input. Specifically, the combination of n Transformers with Broad Learning System (BLS) is as follows: Figure 3As shown, the input to each Transformer layer undergoes internal processing: first, it passes through an Attention mechanism to obtain attention representations between different tokens; then, the multi-head attention results are concatenated and fed into a feedforward neural network FFN1 to obtain representations between different attentions; finally, after residual and normalization (Add & Norm) operations, it is fed into another feedforward neural network FFN2 to obtain deep feature representations; and then, a final residual and normalization operation is performed to obtain the output of the current Transformer layer. This output is used as the input to BLS, combined with the target's true label, and BLS is used to quickly learn the relevance of the transformed features of the current Transformer layer to the adapted natural language understanding task. This design allows for direct use of ridge regression to obtain the output of each BLS layer in a single forward pass. By using a small number of samples as input to a pre-trained language model based on the Transformer architecture, and utilizing the ridge regression learning mechanism of BLS, the output features of each Transformer layer's FFN2 network are used as input to BLS in a single forward pass. Based on the label of the adapted natural language understanding task, the corresponding BLS learning evaluation results are obtained. The specific process is as follows:

[0079] First, assume that the feature of the input text X after being mapped by the word embedding layer is X′, that is...

[0080] X′=XW p-emb (1)

[0081] Among them, W p-emb This refers to the mapping weights of the embedding layer. For a pre-trained language model, these weights are fixed and known. Using the word embedding-mapped features X′ as input to the entire fusion model, the output features h of each Transformer FFN2 network layer are as follows:

[0082] h=Norm((Norm(X′+Attn(X′)W o (2)

[0083] Where Norm(·) represents the normalization operation, W o W1 and W2 are known after the pre-trained language model. Attn(·) represents the attention calculation operation, which is calculated as follows:

[0084]

[0085] Where Q = X′W Q K = X′W K V=X′W V,d k W represents the dimension of vector K. Q W k and W V The parameters of the pre-trained language model are known.

[0086] After obtaining the output features h∈R of the FFN2 network for each Transformer layer... d Then, using this as the input feature of BLS, we first construct the mapping feature Z, as shown below:

[0087] Z=φ(hW e +β e (4)

[0088] Among them W e ∈R d×r and β e ∈R r It is obtained by random sampling from a standard normal distribution, and φ(·) performs the normalization operation. The specific calculation is as follows:

[0089]

[0090] Where d is the hidden layer dimension of the entire enhancement layer features and the mapped features, r represents the rank, and r < d / r. <d,z i That is, the i-th feature element of the mapping feature Z.

[0091] Since randomized weight mapping may interfere with the mapping relationship between input features and labels learned by width learning, and may introduce redundant noise, in order to optimize the constructed mapping features, we borrow the idea of ​​autoencoders and optimize the expressive power of the mapping features by reconstructing the mapping features and adding L1 norm sparsity constraints. The specific reconstruction and optimization objectives are as follows:

[0092]

[0093] The optimization objective can be obtained using the common ADMM iterative algorithm. Therefore, the final mapping layer features can be represented as:

[0094]

[0095] in The mapping weights represent the parameters of the sparse autoencoder, used to map X′ to a mapping feature Z with sparse optimization. s After obtaining the sparsely optimized mapping layer features Z s Then, the enhancement layer node H is constructed using the following calculation method:

[0096] H= ξ(Z s W h +βh (8)

[0097] Among them W h ∈R r×d-r and β h ∈R d-r ξ(·) is a nonlinear activation function obtained by random sampling from a standard normal distribution. In this invention, it is defined as a scaling operation, specifically calculated as follows:

[0098]

[0099] Where h i It is the hth of H i There is one element, where s represents the scaling factor, which is a hyperparameter that is manually set.

[0100] To achieve efficient BLS learning, the hidden layer dimension of the entire enhancement layer features and mapped features is defined as d, which is consistent with the output dimension of each Transformer layer. Simultaneously, low-rank upper and lower random sampling matrices W are utilized. e and W h (with rank r and r << d) to reduce the introduction of intermediate hidden layer nodes in BLS and achieve efficient parameter adaptation.

[0101] After obtaining the enhancement layer features H and the mapping layer features Z s Then, the hidden layer vector is obtained by concatenating the two features. The mapping relationship W between the hidden layer features and the label Y of the adapted natural language understanding task is then learned. Specifically, this learning is achieved by solving the following optimization objective:

[0102]

[0103] Where λ is a hyperparameter that is manually set, representing the penalty term. Limitations on the results.

[0104] This optimization can currently be solved using ridge regression, and the final mapping relationship W can be represented as follows:

[0105] W=(λI+AA T ) -1 A T Y (11)

[0106] Where I is an identity matrix, and A = [Z] s |H]. Finally, the fit between the FFN2 output features of the current Transformer layer and the final adaptation task label, or the evaluation result, of each BLS layer can be expressed as:

[0107] Y eval =AW (12).

[0108] S4. Based on the adaptation results, formulate an appropriate adaptive layer freezing strategy to determine the layers that need to be pre-frozen.

[0109] Assuming the pre-trained language model is composed of n layers of stacked Transformers, let the evaluation result of each BLS layer be denoted as . The pre-frozen layers (pre-frozen layers refer to those parameters that were marked as trainable before training in an n-layer Transformer network, which will be set to non-trainable before or during training, thus eliminating the need to calculate the gradients of trainable parameters) are sorted according to the evaluation results of different layers. The specific steps are as follows:

[0110] S41. Let S be the set of pre-frozen layer sequences, and initialize it as a set of all layer numbers ordered in sequence, S = {1, 2, 3, ..., n}.

[0111] S42. Based on the evaluation results of each layer Rearrange the elements in set S in ascending order to obtain the initial pre-frozen layer sequence S.

[0112] S5. Based on the initial pre-frozen layer sequence S, the complexity of the adaptation task, and the current computing resources, the final frozen layer is determined through an efficient fine-tuning strategy for layer freezing parameters. An efficient fine-tuning module is added to the unfrozen layer to freeze the original model parameters. Only the introduced additional trainable parameters are optimized to achieve an efficient layer freezing fine-tuning process. After the fine-tuning is completed, a vertical domain model that can further solve the corresponding downstream tasks of the fine-tuned data is obtained.

[0113] like Figure 4 As shown, after determining the initial pre-frozen layer sequence S, the final frozen layers are determined based on the complexity of the adapted task, thus achieving efficient parameter fine-tuning based on layer freezing. First, given an adapted natural language understanding task, the final number of frozen layers k is determined according to the difficulty of the task. The specific calculation of k is as follows:

[0114] k = floor(len(S) × α) (13)

[0115] Where floor(·) represents the floor operation, len(·) represents the number of elements in the set, and α is a hyperparameter used to balance model performance and training efficiency. Its value is between [0,1). In this embodiment, α = 0.5 can achieve overall optimality on many natural language understanding tasks, but if only a single adaptation task is considered, its value can fluctuate.

[0116] After determining the final number of frozen layers k, based on the initial pre-frozen layer sequence S obtained in step S42, the layers corresponding to the first k elements of this sequence (i.e., the k layers with the lowest importance) are directly frozen. In the remaining layers (unfrozen layers), a parameter efficient fine-tuning (PEFT) module is added, such as an adapter or a low-rank parameterized matrix (LoRA). The specific function is as follows: Figure 5 As shown in the diagram. Within the Transformer architecture, the input text first passes through the Attention mechanism to obtain the corresponding output. If a LoRA is inserted alongside the Attention mechanism, the input will also pass through the LoRA to obtain the corresponding output, which is then merged with the Attention output as input to the feedforward neural network FFN1. This output is then passed through FFN1. If an Adapter is inserted, the input is sent to the Adapter for processing, and the output undergoes residual + normalization (Add & Normal) operations. Otherwise, residual + normalization is performed directly. The processed output is then used as input to the feedforward neural network FFN2 to obtain the corresponding output. Similarly, if a LoRA is inserted, the input is sent to the LoRA and merged with the original feedforward neural network FFN2 output. If an Adapter is inserted, the original feedforward neural network FFN2 output is processed again by the Adapter to obtain the corresponding output. Finally, a residual + normalization operation is performed again to obtain the final model output.

[0117] Assuming the input text for LoRA or Adapter is x, the specific calculations performed by the Adapter module are as follows:

[0118]

[0119] Among them W down and W up These are the parameters that the Transformer needs to optimize and learn. The nonlinear activation function is represented here. As an example, the common GELU nonlinear activation function is selected in this embodiment, and its calculation is shown below:

[0120] GELU(x)=x*σ(1.702x) (15)

[0121] Where σ(·) represents the sigmoid function. Furthermore, the calculation of LoRA can be expressed as follows:

[0122] LoRA(x)=sABx (16)

[0123] Where s is a hyperparameter, and A and B are parameters that the model needs to optimize and learn. Specifically, the optimization of the Adapter or LoRA parameters is achieved by maximizing the conditional language modeling objective and iteratively updating it using accumulated gradients. The specific conditional language modeling objective can be expressed as follows:

[0124]

[0125] Where x is the input text, y is the output text, Z is the set of input and output text pairs, Φ0 is the parameter of the original pre-trained language model, Θ is the parameter of the additional parameter fine-tuning module that needs to be optimized, p represents the word probability distribution of the predicted current text word; ΔΦ is a whole, representing the parameter fine-tuning module newly added compared to the original pre-trained model; t represents time t or the predicted t-th word; y t This represents the word that should actually be predicted and generated at time t.

[0126] As another example, with the continuous development of efficient parameter fine-tuning methods, variants based on Adapter or LoRA methods are constantly emerging. Therefore, in addition to Adapter or LoRA, other efficient parameter fine-tuning variants can be further covered, such as P-Adapter, Soft Prompt, AdaLoRA, and DoRA. These variants are all embedded within the Transformer architecture, thus ensuring that the execution of the pre-frozen layer strategy and the efficient parameter fine-tuning process combined with layer freezing remain unchanged. Furthermore, different width learning models, such as stacked width learning, employ a similar learning approach to the original width learning, achieving similar results.

[0127] Furthermore, this invention should not be limited to pre-trained language models based on the Transformer architecture. Since width learning and the Transformer architecture have the general characteristic of modality independence, the pre-trained language model based on the Transformer architecture can also be replaced by a large-scale pre-trained visual model or a large-scale multimodal model based on the Transformer architecture.

[0128] The system for implementing the aforementioned method for freezing parameters in a natural language understanding task based on width learning includes:

[0129] Data preprocessing module: preprocesses the model training data to prepare for training the model obtained by fusing wide learning and the Transformer architecture.

[0130] Model architecture fusion module: Design a brand-new joint learning module of width learning and Transformer to prepare for the implementation of the adaptive layer freezing module.

[0131] Adaptive layer freezing module: Based on the data preprocessing module and the model architecture fusion module, it obtains the corresponding pre-frozen layer sequence through a forward computation process using a small amount of training data, preparing for efficient fine-tuning of layer freezing parameters.

[0132] Efficient fine-tuning module for layer freezing parameters: Based on the pre-trained layer sequence obtained by the adaptive layer freezing module, an efficient fine-tuning strategy for layer freezing parameters is designed to achieve the final efficient fine-tuning process for layer freezing based on natural language understanding tasks.

[0133] A computer device according to this embodiment includes: a memory and a processor, and a computer program stored in the memory. When the computer program is executed on the processor, it implements the method for fine-tuning the frozen parameters of the natural language understanding task layer based on width learning.

[0134] This invention combines width learning with a pre-trained language model based on the Transformer architecture and designs a corresponding adaptive pre-freezing layer selection strategy. This strategy can flexibly control the number of pre-frozen layers according to the complexity of the natural language understanding task being adapted, thereby achieving a good balance between the model's adaptation performance and the efficiency of training parameters, making this scheme a unified adaptation framework for natural language understanding tasks.

[0135] For example, in the task of judging the sentiment of movie reviews, the sentiment training data is preprocessed and used as input to the DebertaV3-base pre-trained language model based on the Transformer architecture. Utilizing the ridge regression learning mechanism of BLS, the corresponding BLS learning evaluation results are obtained based on the labels of the adapted natural language understanding task. Based on the adaptation results, an adaptive layer freezing strategy is formulated to determine the pre-frozen layers. According to the pre-frozen layers, the complexity of the adapted task, and the current computing resources, the final frozen layers are determined through an efficient fine-tuning strategy for layer freezing parameters. An efficient parameter fine-tuning module is added to the unfrozen layers to achieve freeze fine-tuning with the introduced additional trainable parameters. After fine-tuning, a vertical domain model capable of further solving downstream tasks corresponding to the fine-tuned data is obtained. For example, the original model can only understand the meaning of text but cannot judge its sentiment or directly output its sentiment. After fine-tuning, the model can output the sentiment category of the text based on the input text. Using the DebertaV3-base pre-trained language model based on the Transformer architecture combined with width learning can achieve better sentiment classification prediction performance than the original method while reducing the number of training parameters by 40%.

[0136] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A method for training a natural language understanding model based on width learning, characterized in that, Includes the following steps: A fusion model is obtained by fusing a pre-trained language model based on the Transformer architecture with width learning; The training data is used as input to a pre-trained language model based on the Transformer architecture. The ridge regression learning mechanism of BLS is used to obtain the evaluation results based on the labels of the adapted natural language understanding task. Based on the evaluation results, develop an appropriate adaptive layer freezing strategy to determine the pre-frozen layer; Based on the pre-frozen layer, the complexity of the adaptation task, and the current computing resources, the final frozen layer is determined through an efficient fine-tuning strategy for layer freezing parameters. An efficient fine-tuning module is added to the unfrozen layer to achieve freeze fine-tuning with the introduced additional trainable parameters, thereby obtaining the vertical domain model of the downstream task corresponding to the fine-tuned data. Developing an appropriate adaptive layer freezing strategy includes the following steps: S41. Assume that the pre-trained language model is... It consists of stacked Transformer layers, and the evaluation result of each BLS layer is denoted as . The pre-frozen layer sequence is obtained by sorting the evaluation results of different layers. S42. Let the set of pre-frozen layer sequences be... Initialized as a set of all layer numbers sorted in order. ; S43. Based on the evaluation results of each layer The elements in set S are rearranged in ascending order to obtain the initial pre-frozen layer sequence; Determining the final frozen layer using an efficient fine-tuning strategy for layer freezing parameters includes the following steps: Determine the final number of layers to freeze based on the given natural language understanding task. : (13) in This indicates the floor function. This indicates that the number of elements in the set is being calculated. This represents the initial pre-frozen layer sequence. These are hyperparameters used to achieve a tradeoff between model performance and training efficiency; the initial pre-frozen layer sequence The former The layer corresponding to each element is the one with the lowest importance. The layers are frozen directly.

2. The method for training a natural language understanding model based on width learning according to claim 1, characterized in that, The fusion method is as follows: Suppose that the pre-trained language model is... The system consists of stacked Transformer layers. The output of each Transformer layer serves as the input to each BLS layer. Training data is input into each Transformer layer, first passing through an Attention mechanism to obtain attention representations between different tokens. The multi-head attention results are then concatenated and fed into a feedforward neural network FFN1 to obtain representations between different attentions. After residual and normalization operations, the data is fed into another feedforward neural network FFN2 to obtain deep feature representations. A final residual and normalization operation is performed to obtain the output of the current Transformer layer. This output is used as the input to the BLS layer. Combined with the target's true label, the BLS layer is used to determine the relevance of the transformed features of the current Transformer layer to the adapted natural language understanding task.

3. The method for training a natural language understanding model based on width learning according to claim 1, characterized in that, Obtaining the evaluation results includes the following steps: First, let's assume the input text is... The features after word embedding layer mapping are: ,Right now (1) in, This refers to the mapping weights of the Embedding layer, which represent the features after word embedding is mapped. As input to the entire fusion model, the output features of the FFN2 network of each Transformer layer... as follows: (2) in This indicates a normalization operation. , , All of these are known after the pre-trained language model. This indicates the attention calculation operation, and the specific calculation is as follows: (3) in , This represents the dimension of vector K. as well as In a pre-trained language model, the parameters are known. After obtaining the output features of the FFN2 network at each Transformer layer... Then, using this as the input feature of BLS, the mapping feature is first constructed. As shown below: (4) in and It is obtained by random sampling from a standard normal distribution. To perform the normalization operation, the specific calculation is as follows: (5) in The hidden layer dimension represents the sum of the features of the entire enhancement layer and the mapped features. Denotes rank, and That is, the first mapping feature Z Each feature element; The expressive power of the mapping features is optimized by reconstructing them and adding L1 norm for sparsity constraints. The specific reconstruction and optimization objectives are as follows: (6) The final mapping layer features are represented as follows: (7) in The mapping weight parameters represent the sparse autoencoder, used to... Mapping to features with sparse optimization ; After obtaining the sparse optimized mapping layer features Then, the enhancement layer node H is constructed using the following calculation method: (8) in and To obtain by random sampling from a standard normal distribution, It is a nonlinear activation function; Enhancement layer features and mapping layer features The hidden layer vectors are obtained by concatenation, and then the appropriate natural language understanding task labels are used. To learn the final mapping relationship between hidden layer features and labels. Specifically, learning is achieved by solving the following optimization objective: (10) in It is a hyperparameter that is manually set, and it is a penalty term. Limitations of results; The solution is obtained through ridge regression, resulting in the final mapping relationship. It can be represented as follows: (11) in It is an identity matrix. ; Finally, the fit between the FFN2 output features of each BLS layer and the final adaptation task label, or the evaluation result, is expressed as follows: (12)。 4. The method for training a natural language understanding model based on width learning according to claim 1, characterized in that, The Parameter Efficient Fine-Tuning (PEFT) module is a low-rank heavy parameterized matrix LoRA. The input training data first passes through the Attention mechanism of the Transformer architecture to obtain the corresponding output. If LoRA is inserted next to the Attention mechanism, the input training data will also pass through LoRA to obtain the corresponding output, and merge it with the output of Attention as the input of the feedforward neural network FFN1. Then, it passes through the feedforward neural network FFN1 to obtain the corresponding output, and the output is used as the input of the feedforward neural network FFN2 to obtain the corresponding output. Similarly, if LoRA is selected to be inserted, the corresponding input will also be sent to LoRA and merged with the original FFN2 output.

5. The method for training a natural language understanding model based on width learning according to claim 1, characterized in that, The Parameter Efficient Fine-Tuning (PEFT) module is an adapter. The input training data first passes through the Attention mechanism of the Transformer architecture to obtain the corresponding output, and then passes through the feedforward neural network FFN1 to obtain the corresponding output. An adapter is inserted after the FFN1 network. Subsequently, the output of FFN1 is sent to the adapter for processing to obtain the corresponding output. After the corresponding output is obtained, if the adapter is selected to be inserted, the output of the original FFN2 is processed by the adapter again to obtain the corresponding output. Finally, a residual + normalization operation is performed again to obtain the final output.

6. A system for implementing the natural language understanding model training method based on width learning as described in claim 1, characterized in that: include: The data preprocessing module preprocesses the training data. The model architecture fusion module is designed with a brand-new width learning and Transformer joint learning module to prepare for the implementation of the adaptive layer freezing module; By integrating the selected pre-trained language model architecture type with the corresponding width learning design, a joint learning architecture for the pre-trained language model is obtained. The adaptive layer freezing module, based on the data preprocessing module and the model architecture fusion module, obtains the corresponding pre-frozen layer sequence through a forward computation process using training data. The efficient fine-tuning module for layer freezing parameters, based on the pre-frozen layer sequence obtained by the adaptive layer freezing module, achieves the final efficient fine-tuning process for layer freezing based on natural language understanding tasks through an efficient fine-tuning strategy for layer freezing parameters.

7. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, the processor causes the processor to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 5.

Images