Sentiment analysis method based on LoRA

Through the combination of LoRA method, TextCNN and attention mechanism, the problem of insufficient efficient fine-tuning and semantic extraction capabilities in the existing technology is solved, and the semantic extraction and generalization of the model is improved without increasing inference time and trainable parameters is achieved. ability.

CN118427344BInactive Publication Date: 2024-11-22HUBEI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410497558.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-24
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing efficient fine-tuning methods based on pre-trained models increase the complexity of the inference stage while increasing the depth of the model. They cannot achieve efficient fine-tuning without increasing the inference time, and cannot fully consider the context information of the text, resulting in strong data dependence. , high labeling costs and limited generalization capabilities.

Method used

The LoRA method is used to simulate the change of the original weight parameters through low-rank matrix decomposition, and introduce a bypass structure to update the weights in the original network. The feature representation module is designed in combination with TextCNN and attention mechanism to ensure that the model does not increase the inference time and trainable parameters. Improve semantic extraction capabilities under the circumstances.

Benefits of technology

This achieves model controllability with the same inference complexity during the fine-tuning process, improves the performance capabilities of semantic extraction and downstream tasks, avoids the problem of deep neural network degradation, and enhances the generalization capabilities of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118427344B_ABST
    Figure CN118427344B_ABST
Patent Text Reader

Abstract

The present invention discloses a sentiment analysis method based on LoRA, which belongs to the field of artificial intelligence technology, including S1, a preprocessing layer, in which data is cleaned and the input sample is segmented using the BERT pre-training model; S2, a word embedding layer, in which the input sample after segmentation is passed through a vector representation layer and mapped into a high-dimensional continuous word vector to capture semantic information; S3, a feature extraction layer, in which the high-dimensional word vector is used as the input of the feature extraction layer to further extract the feature vector; S4, an output layer, in which the generated feature vector is used as the final input sample representation, and the polarity of the text sentiment is judged through the sentiment classification layer. The present invention adopts the above-mentioned sentiment analysis method based on LoRA, and by updating the pre-trained weight matrix using the increment of the low-rank matrix while training the low-rank matrix, the controllable factors of the model are increased without increasing the training parameters and reasoning time, and the problem of excessive dependence of the original LoRA method on the low-rank matrix is ​​alleviated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a sentiment analysis method based on LoRA. Background Art

[0002] In the context of the digital age, sentiment analysis has become a key tool for enterprises and service providers to focus on user experience. It has been widely used in social media analysis, sentiment-driven content recommendation, public opinion monitoring, etc., and not only plays an important role, but also contains huge commercial potential and application prospects.

[0003] Text sentiment analysis can be divided into three levels according to the different granularities of text processing: paragraph-level sentiment analysis, sentence-level sentiment analysis, and aspect-level sentiment analysis. However, whether it is paragraph-level or sentence-level sentiment analysis, they only assume that a text contains only one emotional color, and cannot fully reflect the more detailed emotional expressions of different objects in the text. From the perspective of development at the computational level, it can be divided into dictionary-based statistical methods, machine learning-based methods, traditional deep learning methods, and pre-trained model-based methods. Dictionary-based, machine learning-based, and traditional deep learning methods generally have defects such as strong data dependence, high annotation costs, limited generalization capabilities, and inability to fully consider the contextual information of the text. The present invention is an aspect-level sentiment analysis based on a pre-trained model.

[0004] Most of the existing efficient fine-tuning methods based on pre-trained models are achieved by freezing the pre-trained model and increasing the model depth. Such methods will undoubtedly increase the complexity of the model during the inference stage. Summary of the invention

[0005] The purpose of the present invention is to provide a sentiment analysis method based on LoRA to solve the problems existing in the above-mentioned background technology, and to propose a sentiment analysis method that can achieve efficient fine-tuning without increasing the reasoning time, and has improved performance in semantic extraction and downstream tasks.

[0006] To achieve the above object, the present invention provides a sentiment analysis method based on LoRA, comprising the following steps:

[0007] S1. Clean the data in the preprocessing layer and use the BERT pre-trained model to segment the input samples;

[0008] S2, the input samples after word segmentation in the word embedding layer are passed through the vector representation layer and mapped into high-dimensional continuous word vectors to capture richer semantic information;

[0009] S3, in the feature extraction layer, the high-dimensional word vector is used as the input of the feature extraction layer to further extract a more expressive feature vector;

[0010] S4. In the output layer, the generated feature vector is used as the final input sample representation, and the polarity of the text sentiment is judged through the sentiment classification layer.

[0011] Preferably, in S1, a toolkit is used to segment the input sample to obtain a segmentation result S = [t1, t2, ...t n ], generally a complete word or symbol is used as the segmentation standard.

[0012] Preferably, the specific steps of the word embedding layer of S2 are as follows:

[0013] S21, obtain an original sample input S consisting of n words in S1 = [t1, t2, ...t n ], where t i is the i-th word decomposed according to the word segmentation rule;

[0014] S22. Insert special tags [CLS] and [SEP] at the beginning and end of the original sample input, respectively, to obtain the initial sequence Seq = {[CLS], t1, t2, t3, ... t n ,[SEP]};

[0015] S23, mapping each token and word segment in the initial sequence to an ID according to the vocabulary trained by the BERT pre-trained model, and several IDs constitute the input sequence of the model;

[0016] S24. If the input sequence length is less than the preset length of the model, a special marker is used to fill it to the preset length, and then a segment ID is assigned to each word segment. Usually 0 or 1 is used to distinguish two different sentence segments.

[0017] S25. According to the position of the word in the sentence, a position ID (PositionID) is assigned to each word, providing the model with the position information of the input sample;

[0018] S26. Input the processed original sample input (including input sequence, paragraph information, and position information) into the BERT pre-trained model to obtain the hidden state of each word segmentation, and convert the hidden state output by the model into a word embedding vector for subsequent tasks.

[0019] Preferably, the LoRA method is used in S3 to simulate the change of the original weight parameters through low-rank matrix decomposition. Specifically, a bypass structure is introduced into the original network. The bypass structure consists of a low-rank matrix A and a low-rank matrix B. The increment of the product of the low-rank matrix A and the low-rank matrix B is used as a reference for updating the original weight matrix W. Without increasing additional training parameters and inference time, the controllable elements in the model training are increased, and the training effect of the model is improved. During the training process, the low-rank matrix A and the low-rank matrix B are updated, and the increment of their product is also used to dynamically update the original weight matrix W. The updating process is as follows:

[0020]

[0021] h=W t+1 ·x+A t+1 B t+1 ·x

[0022] Among them, W t is the original weight at the tth time step, A t ,B t are two low-rank matrices at the tth time step, λ is the update ratio of the weight matrix W. If the gradient descent method is used to update W, then W t+1 =W t -ηg(W), where η is the learning rate, g(W) is the gradient of the matrix, and the weight increment of the pre-trained matrix is ​​ΔW = -ηg(W), that is, the full parameter fine-tuning W matrix. The original idea of ​​the LoRA algorithm is to use the multiplication of low-rank matrices A and B to represent the final increment of the original weight matrix W. Therefore, it is feasible to use the increment of the multiplication of A and B as the basis for updating the pre-trained weight matrix W in each step. Through this method, the controllability of the model can be effectively expanded without increasing the number of trainable parameters, so as to enhance the model's ability to extract semantics.

[0023] Preferably, in order to improve the performance of the model on downstream tasks, a feature representation module is designed that captures local features relative to global information under different receptive fields by combining the TextCNN method and the attention mechanism, ensuring that the model can simultaneously focus on local text information and all information at different granularities. After the input sample is embedded and feature extracted by the BERT pre-training model, a feature sentence vector E=(e1,e2,e3,...,e n ), and use it as the input of the current feature representation module. The TextCNN method uses convolution kernels of different sizes to capture local features of different granularities in the text. For a sentence vector X∈R n×k , where n represents the length of the sentence in the sentence vector and K represents the embedding dimension. The calculation process in TextCNN is as follows:

[0024] C i =f(W·x i;i+h-1 +b)

[0025] Where W∈R h×k , h represents the height of the convolution kernel, k represents the width of the convolution kernel, x i;i+h-1 represents the h×k-dimensional local vector formed by concatenating the word vector of the i-th row of the input sentence vector to the word vector of the i+h-1th row, b is the bias term, and f is the nonlinear function;

[0026] Convolution kernel and local vector x i;i+h-1 Perform dot product operation to extract local features C i The convolution kernel slides from top to bottom and multiplies the local vector to obtain the local feature C i to C i-h+1 , and then concatenate them to get C=[C1,C2,...C n-h+1 ] indicates that a convolution operation obtains a feature vector.

[0027] Preferably, the convolution operation can represent phrases or some local information to a certain extent, and cannot focus on the overall information of the sentence. Therefore, after each convolution kernel completes the convolution operation, a multi-head attention mechanism module is connected to inject global information into the local vector to further improve the generalization and representation capabilities of the model. The calculation process of the multi-head attention mechanism is as follows:

[0028]

[0029] h i =Attention(W i (q) q,W i (k) ·k,W i (v) v)

[0030] H=W0·[h1,h2,h3,...,h n ] T

[0031] Among them, W i (q) ,W i (k) ,W i (v) Corresponding to the linear transformation parameter matrices of Query, Key, and Value input vectors, W0 is the linear transformation matrix of the multi-head output splicing vector, h i is the hidden layer output of the i-th attention head, and H is the final output of the multi-head attention mechanism module.

[0032] Preferably, in order to avoid the degradation problem of deep neural network that may occur after the introduction of feature representation module, residual structure is also added. This degradation problem is usually manifested as the difficulty of deep neural network to fully exert its powerful feature extraction ability, resulting in the degradation of model performance. The introduction of residual structure aims to alleviate the problem of gradient vanishing and gradient exploding by directly connecting the outputs between layers, thereby maintaining the depth and complexity of the network. Its final output under the current size of convolution kernel is obtained by the following formula:

[0033] C I =f(x+MutiHeadAttention(TextCNN(x)))

[0034] Among them, x is the input of the TextCNN network layer, f is the nonlinear activation function, and the final output after the current module is as follows:

[0035]

[0036] Among them, I is the number of convolution kernels, C I is the convolution result obtained under the convolution kernel I, and the pooling function is the pooling layer.

[0037] Preferably, in S4, the previously extracted feature vector is converted into the probability distribution of the corresponding classification category through the fully connected layer combined with the softmax function. The specific calculation process is as follows:

[0038] LinerOut L =O L ·W L +b L

[0039] Out = Softmax(LinerOut L )

[0040] class = argmax(Out)

[0041] Among them, O L is the output of the feature extraction layer of the current model, W L is the weight matrix of the fully connected layer, b L It is the bias item of the fully connected layer. After using the Softmax function to obtain the probability of the current word vector corresponding to each category in the classification category, the argmax function is finally used to find the index of the category with the largest probability value, and the index value is mapped to the specific sentiment polarity, which is the model's judgment on the sentiment category of the current input sample.

[0042] Therefore, the present invention adopts the above-mentioned LoRA-based sentiment analysis method, which has the following beneficial effects:

[0043] (1) During the fine-tuning process, the original weights are updated with reference to the incremental low-rank matrix multiplication, increasing the controllable factors while achieving the same inference complexity as full model fine-tuning;

[0044] (2) A TextCNN module combined with an attention mechanism is added to the feature extraction layer to further extract features, ensuring that the model can simultaneously focus on local text information and all information at different granularities, further improving the generalization ability of the model;

[0045] (3) The residual structure is introduced to alleviate the problems of gradient vanishing and gradient exploding by directly connecting the outputs between layers, thereby maintaining the depth and complexity of the network and avoiding the problem of deep neural network degradation that may occur after the introduction of the TextCNN module and the attention mechanism module.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the LoRA network structure of an embodiment of the LoRA-based sentiment analysis method of the present invention;

[0048] Figure 2 Schematic diagram of the updating process of the original weight W of the present invention;

[0049] Figure 3 Schematic diagram of the logical structure of the multi-attention mechanism of the present invention;

[0050] Figure 4 Schematic diagram of the internal residual structure of the model of the present invention;

[0051] Figure 5 It is a schematic diagram of the model structure of the present invention. DETAILED DESCRIPTION

[0052] Example

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] like Figure 1 As shown, a sentiment analysis method based on LoRA includes the following steps:

[0055] S1. Clean the data in the preprocessing layer and use the BERT pre-trained model to segment the input samples;

[0056] The present invention conducts experiments on the SST-2 dataset, Restaurant-2014 dataset, Laptop-2014 dataset, and Twitter dataset.

[0057] In the SemEval2014 task, valid data is first extracted from the data provided by the official website. Among the extracted valid data, the "conflict" label data accounts for less than 3% of all the data, so it is considered as noise and filtered out; for the selected Twitter data set, only special characters such as "@User", "#", etc. are processed;

[0058] The toolkit is used to segment the input sample into words in the form of S = [t1, t2, ... t n ] format. Generally, a complete word or symbol is used as the segmentation standard.

[0059] S2, the input samples after word segmentation in the word embedding layer are passed through the vector representation layer and mapped into high-dimensional continuous word vectors to capture richer semantic information;

[0060] In the preprocessing layer, we obtain an original sample input S=[t1,t2,...t n ], where t i is the i-th word decomposed according to the word segmentation rule;

[0061] Insert [CLS] and [SEP] special tags at the beginning and end of the original sample respectively, and get the initial sequence Seq = {[CLS], t1, t2, t3, ... t n ,[SEP]};

[0062] Map each token and word in the initial sequence to a unique ID based on the vocabulary trained by the BERT pre-trained model. These IDs constitute the input sequence of the model.

[0063] If the input sequence length is less than the preset length of the model, special markers are used to fill it to the preset length, and then a segment ID (SegmentID) is assigned to each word, usually 0 or 1 to distinguish two different sentence segments;

[0064] Then, a position ID (PositionID) is assigned to each word according to its position in the sentence, providing the model with the position information of the input sample;

[0065] Finally, the processed original sample input, including the input sequence, paragraph information, and position information, is input into the BERT pre-trained model to obtain the hidden state of each word segmentation. The hidden state output by the model is converted into a word embedding vector, which can be used for subsequent tasks.

[0066] S3, in the feature extraction layer, the high-dimensional word vector is used as the input of the feature extraction layer to further extract a more expressive feature vector;

[0067] When the downstream task samples are input into the BERT pre-trained model, the word embedding vector obtained through the original BERT pre-trained weights still belongs to the category of word embedding. At this stage, the weights of the BERT model are fixed and are learned from a large amount of unlabeled text. They contain rich language knowledge and can capture the contextual information of the input samples. In the subsequent efficient fine-tuning process or the full model fine-tuning process, the original BERT pre-trained model structure is used to further extract features in combination with the characteristics of the downstream task data distribution.

[0068] The present invention selects the BERT_large version as the baseline research model, and uses the LoRA method as the baseline research method for experiments. The LoRA method simulates the change in the original weight parameters through low-rank matrix decomposition. The specific approach is to introduce a bypass structure into the original network. The bypass structure consists of a downsampling matrix A and an upsampling matrix B. The update amount ΔW of the original weight matrix W is approximated by the product of A and B. In this way, the amount of updateable parameters in the bypass structure will be much smaller than the amount of parameters in the weight matrix W in the original network. During training, LoRA freezes the original parameters of the BERT pre-trained model and only trains the low-rank matrices A and B in the side structure. During the inference process, the original weight parameters are combined with the A and B weights as a substitute for the original weight matrix to achieve efficient fine-tuning without increasing the inference time. The LoRA network structure is as follows Figure 1 As shown, the calculation process is as follows:

[0069] W0+ΔW=W0+A·B

[0070]

[0071] x belongs to the input of the current layer, W0∈R d×k is the original pre-trained weight, which is frozen during the training process and does not accept parameter updates. r×k ,B∈R d×r are two trainable low-rank matrices, and r<<d. When initializing the matrices, random Gaussian initialization is used for matrix A and zero initialization is used for matrix B to ensure that ΔW=A·B is a zero matrix at the beginning of training. ΔW·x is scaled using αγ. Where α is a constant and γ is the rank of the low-rank matrix.

[0072] The LoRA method effectively avoids the problem of increasing the complexity of the original BERT pre-trained model in the inference stage by methods such as the P_Tuning method and Adapter. However, the LoRA method has a strong dependence on low-rank matrices, which makes the number of parameters involved in training very limited, and the incremental update of the low-rank matrix is ​​not enough to learn the knowledge that can complete the downstream task representation. Therefore, the model after the introduction of the LoRA method may show a slight overfitting state during the training process. In order to alleviate the above problems, the present invention refers to the optimization method proposed by the researchers, and uses the increment of the product of the low-rank matrices A and B as a reference for updating the original weight W. Without increasing additional training parameters and inference time, the controllable elements in the model training are increased, thereby improving the training effect of the model. Its logical structure is shown in the figure below. While updating the low-rank matrices A and B during the training process, the training weight matrix W is also dynamically updated using the increment of the product of the AB matrix. The update process of the original weight matrix W is shown as follows. Figure 2 As shown, the hidden layer update process is as follows:

[0073]

[0074] h=W t+1 ·x+A t+1 B t+1 ·x

[0075] Where W t is the original weight at the tth time step, A t ,B t are two low-rank matrices at the tth time step, and λ is the update ratio of the weight matrix W. If the gradient descent method is used to update W, then W t+1 =W t -ηg(W), where η is the learning rate and g(W) is the gradient of the matrix. The weight increment of the pre-trained matrix is ​​ΔW = -ηg(W), which is the full parameter fine-tuning W matrix. The original idea of ​​the LoRA algorithm was to use the product of the low-rank matrices A and B to represent the final increment of the original weight matrix W. Therefore, it is feasible to use the increment of the product of A and B as the basis for updating the pre-trained weight matrix W in each step. Through this method, the controllability of the model can be effectively expanded without increasing the number of trainable parameters, thereby enhancing the model's ability to extract semantics.

[0076] In order to further improve the performance of the model on downstream tasks, this paper combines the TextCNN method and the attention mechanism to design a feature representation module that captures local features relative to global information under different receptive fields, ensuring that the model can simultaneously focus on local text information and all information at different granularities. After the input sample is embedded in the BERT pre-trained model and feature extracted, a feature sentence vector E=(e1,e2,e3,...,e n ), and use it as the input of the current module. The TextCNN method uses convolution kernels of different sizes to capture local features of different granularities in the text. These local features include words, phrases, and parts of sentences. For a sentence vector X∈R n×k , where n represents the length of the sentence in the sentence vector and K represents the embedding dimension. The calculation process in TextCNN is as follows:

[0077] C i =f(W·x i;i+h-1 +b)

[0078] where W∈R h×k , h represents the height of the convolution kernel, and k represents the width of the convolution kernel, which is also equivalent to the word embedding dimension. Unlike images in the CV field, which are used as inputs to CNN, the input in TextCNN is a word matrix composed of word vectors. Therefore, the convolution kernel in TextCNN can only move vertically. i;i+h-1 represents the h×k dimensional local vector formed by concatenating the word vector of the i-th row of the input sentence vector to the word vector of the i+h-1th row, b is the bias term, and f is the nonlinear function. i;i+h-1 Perform dot product operation to extract local features C i The convolution kernel slides from top to bottom and multiplies the local vector to obtain the local feature C i to C i-h+1 , and then concatenate them to get C = [C1, C2, ... C n-h+1 ] indicates that a convolution operation obtains a feature vector.

[0079] The convolution operation can represent phrases or some local information to a certain extent, but cannot focus on the overall information of the sentence. Therefore, after each convolution kernel completes the convolution operation, the multi-head attention mechanism module is connected to inject global information into the local vector to further improve the generalization and representation capabilities of the model. The logical structure of the multi-head attention mechanism is as follows: Figure 3 As shown in , it is essentially a variant of the attention mechanism that uses parallel computing. Each head calculates the attention score independently, but shares the same input. The concatenation of the outputs of multiple heads is the feature value of the local vector after injecting global information. The calculation process is shown in the following formula.

[0080]

[0081] h i =Attention(W i (q) q,W i (k) ·k,W i (v) v)

[0082] H=W0·[h1,h2,h3,...,h n ] T

[0083] Where W i (q) ,W i (k) ,W i (v) They correspond to the linear transformation parameter matrices of the Query, Key, and Value input vectors respectively, and W0 is the linear transformation matrix of the multi-head output concatenation vector. i is the hidden layer output of the i-th attention head, and H is the final output of the multi-head attention mechanism module.

[0084] In order to avoid the problem of deep neural network degradation that may occur after the introduction of the TextCNN module and the attention mechanism module, the present invention introduces a residual structure. This degradation problem is usually manifested as the difficulty of deep neural networks to fully exert their powerful feature extraction capabilities, resulting in a decrease in model performance. The introduction of the residual structure aims to alleviate the problems of gradient vanishing and gradient exploding by directly connecting the outputs between layers, thereby maintaining the depth and complexity of the network. Figure 4 As shown, the final output of the residual structure under the convolution kernel of the current size is given by the following formula:

[0085] C I =f(x+MutiHeadAttention(TextCNN(x)))

[0086] Where x is the input of the TextCNN network layer, f is the nonlinear activation function, and the final output after the current module is as shown in the formula:

[0087]

[0088] Where I is the number of convolution kernels, C I is the convolution result obtained under the convolution kernel I, and the pooling function is the pooling layer.

[0089] S4. In the output layer, the generated feature vector is used as the final input sample representation, and the polarity of the text sentiment is judged through the sentiment classification layer.

[0090] The previously extracted feature vector is converted into the probability distribution of the corresponding classification category through the fully connected layer combined with the softmax function. The specific calculation process is shown in the following formula:

[0091] LinerOut L =O L ·W L +b L

[0092] Out = Softmax(LinerOut L )

[0093] class = argmax(Out)

[0094] Among them, L is the output of the feature extraction layer of the current model, W L is the weight matrix of the fully connected layer, b L is the bias term of the fully connected layer. After obtaining the probability of the current word vector corresponding to each category in the classification category with the help of the Softmax function, the argmax function is finally used to find the index of the category with the largest probability value, and the index value is mapped to the specific sentiment polarity, which is the model's judgment on the sentiment category of the current input sample.

[0095] Model training:

[0096] The loss function is an important part of machine learning and deep learning. Its main function is to measure the difference or error between the model's predicted output and the true label. It promotes model learning by minimizing the loss value and improves the model's fit to downstream task data. The present invention selects the cross entropy loss function (CrossEntropyLoss) as the method for calculating the loss value, and its calculation method is shown in the following formula:

[0097]

[0098] The Cross function is the cross entropy loss function, y is the sentiment category of the sample annotation, is the emotion category predicted by the model. To prevent the model from overfitting, an L2 regularization term is added on the basis of the cross entropy function. The model parameters are penalized by adding the square of the weight in the loss function, which limits the size of the model parameters and makes the model tend to learn simple patterns, thereby improving the generalization performance of the model. λ1 is the L2 regularization coefficient, and θ is a trainable parameter. Finally, the present invention uses the Adam optimizer to update the trainable parameters. To ensure the rationality of the model performance evaluation, the accuracy and weighted F1 value are used as evaluation indicators.

[0099] Comparative experiment:

[0100] In order to verify the effectiveness of the model, several baseline models are introduced for comparison. The comparison results are as follows:

[0101]

[0102] (1) Glove_BiLSTM_TextCNN: Glove word vectors are used as the word embedding layer, and a bidirectional LSTM structure is used to capture the long-term dependencies of sentences. The word vectors after bidirectional LSTM are then injected into the TextCNN network to extract local features of different sizes.

[0103] (2) BERT_large: Based on the Transformer architecture, it uses a deeper network structure, including a 24-layer Transformer encoder and 16 attention heads.

[0104] (3) BERT_TextCNN: BERT_large is used as the word embedding layer and combined with TextCNN to focus on local information.

[0105] (4) P_tuningv2: Use the hint learning method of P_Tuningv2 to perform efficient fine-tuning based on BERT_large.

[0106] (5) LoRA: Apply the LoRA method on the large version of the BERT model to achieve efficient fine-tuning.

[0107] (6) LoRA_O: Update the original weight parameters while updating the low-rank matrix, and introduce this method into the BERT_large model for experiments.

[0108] (7) This model (such as Figure 5 As shown in Figure 5), based on model (6), the TextCNN network structure and residual network structure with added attention mechanism are introduced.

[0109] From the comparative experimental results, we can see that:

[0110] The improved LoRA method LoRA_O has better performance than the original LoRA method in all experimental data sets, proving that the method of using low-rank matrix increments to update the original weight parameters can more effectively improve the performance of the model than the method of only using low-rank matrix multiplication to approximate the original matrix increments.

[0111] By comparing the performance of BERT_large and BERT_TextCNN on the current dataset, it is found that the local features extracted by TextCNN have an enhanced effect on the overall feature vector representation, which helps to enhance the model's semantic understanding of the input samples. This verifies the necessity of introducing TextCNN into the model.

[0112] By adding a TextCNN network with an integrated attention mechanism after the BERT pre-trained model, we achieved better performance than the original model on the three current datasets. This shows that paying attention to global information at the granularity of phrases or words can capture deeper semantic information, thereby improving the performance of the model.

[0113] Hyperparameter Ablation:

[0114] 1. Update coefficient λ of original weight matrix

[0115] When improving the LoRA method, it involves the design of the original weight matrix increment. λ is the ratio of the original weight matrix update. Therefore, under the condition of fixing other hyperparameters, the present invention explores the impact of the change of λ value in the range of 0.5 to 3.0 on the model performance. The experimental results are shown in the following table:

[0116] SST2 Restaurant2014 Laptop2014 Twitter 0.5 93.81 89.31 80.56 74.85 1.0 94.15 89.41 80.56 74.96 1.5 94.27 89.91 81.58 75.14 2.0 94.04 89.00 80.56 74.70 2.5 93.92 90.03 80.56 75.59 3.0 93.92 89.41 81.03 75.10

[0117] From the above experimental results, it can be seen that when the λ value is 1.5, the model performs best overall on the current dataset. A lower λ value may cause the original weight matrix to fail to fully update to the optimal state before the model overfits, while a higher λ value may cause the model to over-adjust when searching for the optimal parameters, thereby missing the optimal solution.

[0118] 2. Rank γ and scaling factor α of low-rank matrix

[0119] In order to understand the role of the rank γ and scaling factor α of the low-rank matrix on the model classification effect, this experiment explored the impact of different values ​​of γ and α on the model classification performance while keeping other hyperparameters unchanged. The results are as follows:

[0120] SST2 Restaurant2014 Laptop2014 Twitter γ=2 α=4 93.81 88.90 80.56 75.48 γ=4 α=8 94.15 88.90 80.72 74.41 γ=8 α=16 94.15 88.90 81.19 74.26 γ=16 α=32 94.27 89.91 81.58 75.14 γ=32 α=64 93.92 89.52 82.13 74.56

[0121] By observing the performance of the model under different combinations of ranks and scaling factors, it can be found that when γ is 16 and α is 32, the overall performance of the model on the current dataset is the best. When the rank of the low-rank matrix is ​​too low, the number of model parameters may not be enough to capture the key knowledge of downstream tasks. As the rank increases, the number of learnable parameters of the model increases, and the model can also capture more features. When the rank and scaling factor increase to a larger value, the incremental knowledge content increases, the overall scaling factor remains unchanged, and the model is prone to overfitting.

[0122] Therefore, the present invention adopts the above-mentioned sentiment analysis method based on LoRA. By using the increment of the low-rank matrix to update the pre-trained weight matrix while training the low-rank matrix, the controllable factors of the model are increased without increasing the training parameters and reasoning time, thereby alleviating the problem of excessive dependence of the original LoRA method on the low-rank matrix. Then, the TextCNN network structure with an attention mechanism is introduced to ensure that the model can simultaneously pay attention to local text information and all information at different granularities, thereby improving the semantic extraction ability of the model in word or phrase structure. And the residual structure is introduced to alleviate the problems of gradient vanishing and gradient exploding by directly connecting the outputs between layers, thereby maintaining the depth and complexity of the network.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A sentiment analysis method based on LoRA, It is characterized in that The following steps are involved: S1. Clean the data in the preprocessing layer and use the BERT pre-trained model to segment the input samples; S2, the input sample after word segmentation in the word embedding layer passes through the vector representation layer and is mapped into a high-dimensional continuous word vector to capture semantic information; S3, in the feature extraction layer, the high-dimensional word vector is used as the input of the feature extraction layer to further extract the feature vector; S4, in the output layer, the generated feature vector is used as the final input sample representation, and the polarity of the text sentiment is judged through the sentiment classification layer; S3 uses the LoRA method to simulate the change of the original weight parameters through low-rank matrix decomposition. Specifically, a bypass structure is introduced into the original network. The bypass structure consists of a low-rank matrix A and a low-rank matrix B. The increment of the product of the low-rank matrix A and the low-rank matrix B is used as a reference for updating the original weight W. During the training process, the low-rank matrix A and the low-rank matrix B are updated while the increment of their product is also used to dynamically update the original training weight W. The update process is as follows: h=W t+1 ·x+A t+1 B t+1 ·x Among them, W t is the original weight at the tth time step, A t ,B t are two low-rank matrices at the tth time step, λ is the update ratio of the weight matrix W. If the gradient descent method is used to update W, then W t+1 =W t -ηg(W), where η is the learning rate and g(W) is the gradient of the matrix. The weight increment of the pre-trained matrix is ​​ΔW = -ηg(W).

2. The LoRA-based sentiment analysis method according to claim 1, Features: In S1, the toolkit is used to segment the input sample to obtain the segmentation result S = [t 1 ,t 2 ,...t n ].

3. The LoRA-based sentiment analysis method according to claim 1, It is characterized in that The specific steps of the word embedding layer of S2 are as follows: S21, obtain an original sample input S consisting of n words in S1 = [t 1 ,t 2 ,...t n ], where t i is the i-th word decomposed according to the word segmentation rule; S22. Insert special tags [CLS] and [SEP] at the beginning and end of the original sample input, respectively, to obtain the initial sequence Seq = {[CLS], t 1 ,t 2 ,t 3 ,...t n ,[SEP]}; S23, mapping each token and word segment in the initial sequence to an ID according to the vocabulary trained by the BERT pre-trained model, and several IDs constitute the input sequence of the model; S24, if the input sequence length is less than the preset length of the model, a special marker is used to fill it to the preset length, and then a segment ID is assigned to each word segment; S25. According to the position of the word in the sentence, a position ID is assigned to each word, providing the model with the position information of the input sample; S26. Input the processed original samples into the BERT pre-trained model to obtain the hidden state of each word segmentation, and convert the hidden state output by the model into a word embedding vector for subsequent tasks.

4. The LoRA-based sentiment analysis method according to claim 1, Features: In order to improve the performance of the model in downstream tasks, a feature representation module is designed by combining the TextCNN method and the attention mechanism. After the input sample is embedded and feature extracted by the BERT pre-training model, a feature sentence vector E with text expression ability is obtained. 1 ,e 2 ,e 3 ,...,e n ), and use it as the input of the current feature representation module. The TextCNN method uses convolution kernels of different sizes to capture local features of different granularities in the text. For a sentence vector X∈R n×k , where n represents the length of the sentence in the sentence vector and k represents the embedding dimension. The calculation process in TextCNN is as follows: C i =f(W·x i;i+h-1 +b) Where W∈R h×k , h represents the height of the convolution kernel, k represents the width of the convolution kernel, x i;i+h-1 represents the h×k-dimensional local vector formed by concatenating the word vector of the i-th row of the input sentence vector to the word vector of the i+h-1th row, b is the bias term, and f is the nonlinear function; Convolution kernel and local vector x i;i+h-1 Perform dot product operation to extract local features C i The convolution kernel slides from top to bottom and multiplies the local vector to obtain the local feature C i to C i-h+1 , and then concatenate them to get C=[C 1 ,C 2 ,...C n-h+1 ] indicates that a convolution operation obtains a feature vector.

5. The LoRA-based sentiment analysis method according to claim 1, It is characterized in that After each convolution kernel completes the convolution operation, the multi-head attention mechanism module is connected to inject global information into the local vector. The calculation process of the multi-head attention mechanism is as follows: h i =Attention(W i (q) ·q,W i (k) ·k,W i (v) ·v) H=W 0 ·[h 1 ,h 2 ,h 3 ,...,h n ] T Among them, W i (q) ,W i (k) ,W i (v) Corresponding to the linear transformation parameter matrices of Query, Key, and Value input vectors, W 0 is the linear transformation matrix of the multi-head output splicing vector, h i is the hidden layer output of the i-th attention head, and H is the final output of the multi-head attention mechanism module.

6. The LoRA-based sentiment analysis method according to claim 1, It is characterized in that A residual structure is added after the feature representation module, and its final output under the convolution kernel of the current size is obtained as follows: C I =f(x+MutiHeadAttention(TextCNN(x))) Among them, x is the input of the TextCNN network layer, f is the nonlinear activation function, and the final output after the current module is as follows: Among them, I is the number of convolution kernels, C I is the convolution result obtained under the convolution kernel I, and the pooling function is the pooling layer.

7. The LoRA-based sentiment analysis method according to claim 1, It is characterized in that In S4, the previously extracted feature vector is converted into the probability distribution of the corresponding classification category through the fully connected layer combined with the softmax function. The specific calculation process is as follows: LinerOut L =O L ·W L +b L Out=Softmax(LinerOut L ) class = argmax(Out) Among them, O L is the output of the feature extraction layer of the current model, W L is the weight matrix of the fully connected layer, b L It is the bias item of the fully connected layer. After using the Softmax function to obtain the probability of the current word vector corresponding to each category in the classification category, the argmax function is finally used to find the index of the category with the largest probability value, and the index value is mapped to the specific sentiment polarity, which is the model's judgment on the sentiment category of the current input sample.