A text stance detection method based on efficient fine-tuning of open source large language models

Through LoRA efficient fine-tuning method, the problems of low accuracy and high training cost of pre-trained large language models in professional fields are solved, and efficient position detection in professional fields is achieved, privacy leakage and high latency are avoided, and the wide application of the model is improved. and cost-effective.

CN118734848BActive Publication Date: 2024-11-12NO 30 INST OF CHINA ELECTRONIC TECH GRP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411216935.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-12
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

The existing pre-trained large language models have low accuracy and high training costs in professional field position detection, and there are privacy leaks and high latency problems for unopen source models.

Method used

Using LoRA efficient fine-tuning method, by freezing some parameters of the model, only fine-tuning of the low-rank decomposition matrix of the training layer is fine-tuned, and a text position detection method based on the open source large language model is constructed to ensure that data processing is within the user's control range and reduce computing resources. rely.

Benefits of technology

It improves the accuracy and stability of the model's position in the professional field, reduces training costs and delays, avoids privacy leakage and high latency problems, and makes the large language model more widely used and practically feasible in the professional field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118734848B_ABST
    Figure CN118734848B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of text analysis, and provides a method for detecting text stance based on an efficient fine-tuned open source large language model, comprising the following steps: step 1, constructing prompt words and sentences through prompt word engineering; step 2, constructing and training a fine-tuned open source large language model; step 3, inputting the prompt words and sentences into the fine-tuned open source large language model for text stance detection. The present invention improves the stance detection accuracy and stability of the model in professional fields. The training process of the model is optimized, and the dependence of the stance detection method based on the pre-trained open source large language model on computing resources when applied in specific professional fields is significantly reduced, thereby improving efficiency and cost-effectiveness, and making the deployment of the stance detection method based on the large language model in professional fields more extensive and practical.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of text analysis, and in particular to a text stance detection method based on an efficient fine-tuning open source large language model. Background Art

[0002] At present, the commonly used stance detection methods include stance detection based on machine learning, stance detection based on deep learning, and stance detection based on pre-trained language models. Among them, the stance detection method based on pre-trained large language models has the best performance because it can use the deep language understanding ability learned from large-scale data. Representative models include Generative Pre-Trained (GPT, a deep learning model based on the Internet, trained with available data, and text generation). However, models such as GPT-4 (a language model released by OpenAI, an American artificial intelligence research laboratory) and Wenxin Yiyan are not open source and require payment, and there are problems such as privacy leakage and high latency. In addition, although the existing pre-trained large language models are pre-trained on large-scale data sets, they lack prior knowledge related to stance detection in professional fields, so the accuracy is low in stance detection applications in professional fields (such as social media analysis, attitudes towards events, etc.). It is usually necessary to fine-tune them to learn prior knowledge in professional fields, thereby improving the accuracy of large language models in downstream tasks. However, traditional fine-tuning methods usually require updating all parameters of the model, which consumes a lot of computing resources, resulting in high actual application costs. Summary of the invention

[0003] The present invention aims to provide a text stance detection method based on an efficient fine-tuning open source large language model to solve the following two problems:

[0004] (1) Solve the problems of common large language models that are not open source, such as GPT-4, which require payment, have privacy leaks and high latency.

[0005] (2) Solve the problem that the existing pre-trained large language models lack prior knowledge related to stance detection, resulting in low accuracy, and solve the problem that traditional fine-tuning methods are difficult to train, have high training costs and low efficiency.

[0006] The present invention provides a text stance detection method based on an efficient fine-tuning open source large language model, comprising the following steps:

[0007] Step 1, constructing prompt words and sentences through prompt word engineering;

[0008] Step 2: Build and train a fine-tuned open source large language model;

[0009] Step 3: Input the prompt words and sentences into the fine-tuned open source large language model for text stance detection.

[0010] Furthermore, in step 1, the text to be detected is and targets that require stance detection Automatically fill in the prompt word to form a prompt sentence .

[0011] Furthermore, step 2 includes the following sub-steps:

[0012] Step 2.1, model initialization, includes:

[0013] Selecting an open source large language model , and identify the trainable layers in the attention module;

[0014] Freeze the parameters of all layers except the trainable layers to obtain a parameter-frozen model ;

[0015] Freeze the model in parameters Add the LoRA adaptation layer to the trainable layer and initialize the two low-rank decomposition matrices A and B in the LoRA adaptation layer to obtain the adaptation layer initialization model ;

[0016] Step 2.2, fine-tune text preprocessing, including:

[0017] First, the professional field fine-tuning text set T is input into the text preprocessing module, and after data format conversion, a json dataset D supporting LoRA fine-tuning is obtained;

[0018] Then input the json dataset D into the text segmentation module, and after the dataset is segmented, n batches of datasets are obtained. , each batch of data sets Contains H batches of input data And batch corresponding label ;

[0019] Step 2.3, LoRA fine-tuning, includes:

[0020] The dataset Batch input data Recurrent Input Adaptation Layer Initialization Model The LoRA efficient fine-tuning open source large language module composed of the LoRA fine-tuning module is propagated through the forward network to obtain batch prediction data ;

[0021] Batch prediction data And batch corresponding label Input the multi-classification cross entropy loss calculation module, and after the loss function calculation, get the loss function value ;

[0022] The loss function value and optimizer Input the back propagation and gradient update module, calculate the gradient and back propagate to update the two low-rank decomposition matrices A and B in the LoRA adaptation layer, and iterate n times to get the fine-tuned open source large language model. .

[0023] Furthermore, the method for initializing the two low-rank decomposition matrices A and B in the LoRA adaptation layer in step 2.1 is expressed as:

[0024]

[0025] in, , express OK The real matrix space of columns, Respectively OK The real matrix space of columns, is a hypothetical intrinsic rank during the weight update process, z is the input feature dimension of the attention module, is the output feature dimension of the attention module, is the output vector of the original attention module, is the output vector of the attention module after adding the LoRA adaptation layer.

[0026] Furthermore, the loss function value in step 2.3 is The calculation method is as follows:

[0027]

[0028] in, Is the batch corresponding label No. The one-hot encoding vector of samples, is the batch prediction data No. One-hot encoding vector of samples.

[0029] Furthermore, step 3 includes the following sub-steps:

[0030] Step 3.1, input processing, includes:

[0031] According to the predefined word list Perform word segmentation to obtain a phrase list ;

[0032] List of phrases Each ID vector Perform word embedding mapping to obtain word embedding vector ;

[0033] Step 3.2, fine-tune the open source large language model processing, including:

[0034] Embedding words into vectors Input fine-tuning of open source large language models , after RMS normalization, we get the normalized vector ;

[0035] Normalize the vector Input the fine-tuned attention module, and after mapping, obtain the query vector , key vector and a numeric vector ;

[0036] The query vector , key vector and a numeric vector Input the core attention operation module, and after attention operation processing, obtain the weighted vector ;

[0037] The weight vector Input the multi-layer perceptron module, and after multi-layer perceptron operation processing, obtain the feature vector ;

[0038] Step 3.3, output processing, includes:

[0039] The feature vector The input and output processing module is normalized by RMS to obtain the normalized vector ;

[0040] Will Input word embedding inverse operation module, after decoder restoration processing, obtain probability score vector , the probability score vector Represents each ID vector The probability score of

[0041] The probability score vector Input index prediction module, after Operation, select the ID vector with the highest probability in the probability distribution As output, we get a character index vector ;

[0042] Convert character index vector Input vocabulary conversion module, after matching with predefined vocabulary, get the next character predicted by the model ;

[0043] character The prompt statement will be the same as the original input Merge as input for the next loop , continue to input fine-tune the open source large language model Process and repeat the operation until the stance detection result is generated until.

[0044] Furthermore, in step 3.1, the word embedding mapping formula is expressed as:

[0045]

[0046] in, is the word embedding vector, is the word embedding matrix, is a phrase list The ID vector in .

[0047] Furthermore, in step 3.2, the query vector , key vector and a numeric vector The calculation method is as follows:

[0048]

[0049] in, is the corresponding query vector The weight matrix of the mapping process, is the corresponding query vector The bias matrix of the mapping process; is the corresponding key vector The weight matrix of the mapping process, is the corresponding key vector The bias matrix of the mapping process; is the corresponding numerical vector The weight matrix of the mapping process, is the corresponding numerical vector The bias matrix for the mapping process.

[0050] Furthermore, in step 3.2, the weighted vector The calculation method is as follows:

[0051]

[0052] in, is the attention operation processing, is the probability weight distribution process, is the key vector The transpose of is the key vector Dimension.

[0053] Furthermore, in step 3.2, the feature vector The calculation method is as follows:

[0054]

[0055] Among them, among them, is the linear mapping matrix in the linear transformation parameters that expands the vector dimension by four times, is the bias matrix in the linear transformation parameters that expands the vector dimension by four times, is the linear mapping matrix in the linear transformation parameters that reduces the vector dimension by four times, is the bias matrix in the linear transformation parameters that reduces the vector dimension by a factor of four, is a non-linear activation function.

[0056] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0057] The present invention provides a text stance detection method based on a LoRA efficient fine-tuning open source large language model that can be deployed locally and privately, ensuring that data processing does not leave the scope of user control, avoiding network delays, improving response speed, and solving the privacy leakage and high latency problems of existing online large language models, while improving the stance detection accuracy and stability of the model in professional fields. In addition, the training process of the model is optimized, and a stance detection model is constructed based on the LoRA efficient fine-tuning method. This method only trains a small number of parameters in the model, which significantly reduces the dependence on computing resources compared to the traditional large language model full fine-tuning method, improves efficiency and cost-effectiveness, and makes the training and deployment of stance detection methods based on large language models in professional fields more extensive and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of a text stance detection method based on an efficient fine-tuning open source large language model in an embodiment of the present invention.

[0059] Figure 2 This is a flowchart of constructing and training a fine-tuned open source large language model in an embodiment of the present invention.

[0060] Figure 3 This is a flow chart of inputting prompt words and sentences into a fine-tuned open source large language model to perform text stance detection in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Example

[0064] like Figure 1 As shown, this embodiment proposes a text stance detection method based on an efficient fine-tuning open source large language model, comprising the following steps:

[0065] Step 1: Construct prompt words and sentences through prompt word engineering

[0066] This method sets the prompt word as "Zhang San made a statement: {text input}, is Zhang San's position on {target} supportive, opposed or neutral?". The text input is the text in the test data that needs to be detected. , the target is the target that needs position detection . The text that needs stance detection and targets that require stance detection Automatically fill in the prompt word to form a prompt sentence .

[0067] Step 2: Build and train a fine-tuned open source large language model

[0068] The professional domain fine-tuning text set T is input into the LoRA (Low-Rank Adaptation) efficient fine-tuning open source large language module to obtain After the iteration, fine-tune the open source language model, such as Figure 2 As shown, it specifically includes three steps: model initialization, fine-tuning text preprocessing and LoRA fine-tuning.

[0069] Step 2.1, model initialization

[0070] Parameter freezing: First choose an open source large language model , and identify the mapping matrix in the attention module Then freeze the parameters of all layers except the trainable layers to obtain the parameter frozen model .

[0071] Adaptation layer parameter initialization: freeze the model in parameters Add a LoRA adaptation layer to the trainable layer and decompose the two low-rank matrices in the LoRA adaptation layer Perform initialization processing to obtain the adaptation layer initialization model , the specific calculation method is as follows:

[0072]

[0073] in, , express OK The real matrix space of columns, Respectively OK The real matrix space of columns, is a hypothetical intrinsic rank during the weight update process, z is the input feature dimension of the attention module, is the output feature dimension of the attention module, is the output vector of the original attention module, is the output vector of the attention module after adding the LoRA adaptation layer.

[0074] Step 2.2, fine-tune text preprocessing

[0075] First, the professional field fine-tuning text set T is input into the text preprocessing module. After data format conversion, a json dataset D supporting LoRA fine-tuning is obtained. The dataset format is as follows:

[0076] {“conversations”: [{

[0077] “role”: “user”,

[0078] “content”: “Zhang San made a statement: {text input}\nDoes Zhang San support, oppose or is he neutral about {target}?”

[0079] },{

[0080] “role”: “assistant”

[0081] “content”: “neutral”

[0082] }]}

[0083] Then input the json dataset D into the text segmentation module, and after the dataset is segmented, n batches of datasets are obtained. , each batch of data sets Contains H batches of input data And batch corresponding label .

[0084] Step 2.3, LoRA fine-tuning

[0085] First, the dataset Batch input data Recurrent Input Adaptation Layer Initialization Model The LoRA efficient fine-tuning open source large language module composed of the LoRA fine-tuning module is propagated through the forward network to obtain batch prediction data .

[0086] Then batch prediction data And batch corresponding label Input the multi-classification cross entropy loss calculation module, and after the loss function calculation, get the loss function value , the specific calculation method is as follows:

[0087]

[0088] in, Is the batch corresponding label No. The one-hot encoding vector of samples, is the batch prediction data No. The one-hot encoding vector of samples, if and No. The first element is 1, which means and Belong to Tag categories, and There is only one element that is 1, and the rest are 0.

[0089] Finally, the loss function value and optimizer Input back propagation and gradient update module, after calculating gradient and back propagation, update the two low-rank decomposition matrices in LoRA adaptation layer After n iterations, we get a fine-tuned open source large language model .

[0090] Step 3: Input the prompt words into the fine-tuned open source large language model for text stance detection

[0091] Prompt words Input fine-tuning of open source large language models , the output is in text On target The determined position ,like Figure 3 As shown, it specifically includes three steps: input processing, fine-tuning of the open source large language model processing, and output processing.

[0092] Step 3.1, input processing

[0093] First, the prompt words and sentences are sorted according to the predefined word list. Perform word segmentation to obtain a phrase list .

[0094] Then the phrase list Each ID vector Perform word embedding mapping to obtain word embedding vector The word embedding mapping formula is expressed as:

[0095]

[0096] in, is the word embedding vector, is the word embedding matrix, is a phrase list The ID vector in .

[0097] Step 3.2: Fine-tune the open source large language model processing

[0098] First, embed the word vector Input fine-tuning of open source large language models , after RMS normalization, we get the normalized vector , the specific calculation method is as follows:

[0099]

[0100] in, is the word embedding vector No. dimension, is the normalized vector No. dimension, is the word embedding vector Dimension.

[0101] Then normalize the vector Input the fine-tuned attention module, and after mapping, obtain the query vector , key vector and a numeric vector , the specific calculation method is as follows:

[0102]

[0103] in, is the corresponding query vector The weight matrix of the mapping process, is the corresponding query vector The bias matrix of the mapping process; is the corresponding key vector The weight matrix of the mapping process, is the corresponding key vector The bias matrix of the mapping process; is the corresponding numerical vector The weight matrix of the mapping process, is the corresponding numerical vector The bias matrix for the mapping process.

[0104] Next, the query vector , key vector and a numeric vector Input the core attention operation module, and after attention operation processing, obtain the weighted vector , the specific calculation method is as follows:

[0105]

[0106] in, is the attention operation processing, is the probability weight distribution process, is the key vector The transpose of is the key vector Dimension.

[0107] Finally, the weight vector Input the multi-layer perceptron module, and after MLP (Multilayer Perceptron) operation processing, obtain the feature vector , the specific calculation method is as follows:

[0108]

[0109] in, is the linear mapping matrix in the linear transformation parameters that expands the vector dimension by four times, is the bias matrix in the linear transformation parameters that expands the vector dimension by four times, is the linear mapping matrix in the linear transformation parameters that reduces the vector dimension by four times, is the bias matrix in the linear transformation parameters that reduces the vector dimension by a factor of four, is a non-linear activation function.

[0110] Step 3.3, output processing

[0111] First, the feature vector The input and output processing module is normalized by RMS to obtain the normalized vector , the specific calculation method is as follows:

[0112]

[0113] in, is the eigenvector No. dimension, is the normalized vector No. dimension, is the eigenvector Dimension.

[0114] Then Input word embedding inverse operation module, after decoder restoration processing, obtain probability score vector , the specific calculation method is as follows:

[0115]

[0116] Among them, the probability score vector Represents each ID vector The probability score of is the mapping matrix restored by the decoder, is the bias vector.

[0117] Then the probability score vector Input index prediction module, after Operation, select the ID vector with the highest probability in the probability distribution As output, we get a character index vector .

[0118] Then the character index vector Input vocabulary conversion module, after matching with predefined vocabulary, get the next character predicted by the model .

[0119] Last predicted character The prompt statement will be the same as the original input Merge as input for the next loop , continue to input fine-tune the open source large language model Process and repeat this operation until the stance detection result is generated until.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A text stance detection method based on an efficient fine-tuned open source large language model, characterized in that: The steps include: Step 1, constructing prompt words and sentences through prompt word engineering; Step 2: Build and train a fine-tuned open source large language model; Step 3: Input the prompt words and sentences into the fine-tuned open source large language model to perform text stance detection; Step 2 includes the following sub-steps: Step 2.1, model initialization, includes: Selecting an open source large language model , and identify the trainable layers in the attention module; Freeze the parameters of all layers except the trainable layers to obtain a parameter-frozen model ; Freeze the model in parameters Add a LoRA adaptation layer to the trainable layer and decompose the two low-rank matrices in the LoRA adaptation layer Perform initialization processing to obtain the adaptation layer initialization model ; Step 2.2, fine-tune text preprocessing, including: The professional field fine-tuning text set T is input into the text preprocessing module, and after data format conversion, a json dataset D supporting LoRA fine-tuning is obtained; Input the json dataset D into the text segmentation module, and after the dataset is segmented, n batches of datasets are obtained. , each batch of data sets Contains H batches of input data And batch corresponding label ; Step 2.3, LoRA fine-tuning, includes: The dataset Batch input data Recurrent Input Adaptation Layer Initialization Model The LoRA efficient fine-tuning open source large language module composed of the LoRA fine-tuning module is propagated through the forward network to obtain batch prediction data ; Batch prediction data And batch corresponding label Input the multi-classification cross entropy loss calculation module, and after the loss function calculation, get the loss function value ; The loss function value and optimizer Input back propagation and gradient update module, after calculating gradient and back propagation, update the two low-rank decomposition matrices in LoRA adaptation layer After n iterations, we get a fine-tuned open source large language model .

2. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 1, characterized in that: The method for initializing the two low-rank decomposition matrices A and B in the LoRA adaptation layer in step 2.1 is expressed as: in, , express OK The real matrix space of columns, Respectively OK The real matrix space of columns, is a hypothetical intrinsic rank during the weight update process, z is the input feature dimension of the attention module, is the output feature dimension of the attention module, is the output vector of the original attention module, is the output vector of the attention module after adding the LoRA adaptation layer.

3. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 1, characterized in that: The loss function value in step 2.3 The calculation method is as follows: in, Is the batch corresponding label No. The one-hot encoding vector of samples, is the batch prediction data No. One-hot encoding vector of samples.

4. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1, input processing, includes: According to the predefined word list Perform word segmentation to obtain a phrase list ; List of phrases Each ID vector Perform word embedding mapping to obtain word embedding vector ; Step 3.2, fine-tune the open source large language model processing, including: Embedding words into vectors Input fine-tuning of open source large language models , after RMS normalization, we get the normalized vector ; Normalize the vector Input the fine-tuned attention module, and after mapping, obtain the query vector , key vector and a numeric vector ; The query vector , key vector and a numeric vector Input the core attention operation module, and after attention operation processing, obtain the weighted vector ; The weight vector Input the multi-layer perceptron module, and after multi-layer perceptron operation processing, obtain the feature vector ; Step 3.3, output processing, includes: The feature vector The input and output processing module is normalized by RMS to obtain the normalized vector ; Will Input word embedding inverse operation module, after decoder restoration processing, obtain probability score vector , the probability score vector Represents each ID vector The probability score of The probability score vector Input index prediction module, after Operation, select the ID vector with the highest probability in the probability distribution As output, we get a character index vector ; Convert character index vector Input vocabulary conversion module, after matching with predefined vocabulary, get the next character predicted by the model ; character The prompt statement will be the same as the original input Merge as input for the next loop , continue to input fine-tune the open source large language model Process and repeat the operation until the stance detection result is generated until.

5. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 4, characterized in that: In step 3.1, the word embedding mapping formula is expressed as: in, is the word embedding vector, is the word embedding matrix, is a phrase list The ID vector in .

6. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 4, characterized in that: In step 3.2, the query vector , key vector and a numeric vector The calculation method is as follows: in, is the corresponding query vector The weight matrix of the mapping process, is the corresponding query vector The bias matrix of the mapping process; is the corresponding key vector The weight matrix of the mapping process, is the corresponding key vector The bias matrix of the mapping process; is the corresponding numerical vector The weight matrix of the mapping process, is the corresponding numerical vector The bias matrix for the mapping process.

7. The method for text stance detection based on an efficient fine-tuned open source large language model according to claim 4, characterized in that: In step 3.2, the weighted vector The calculation method is as follows: in, is the attention operation processing, is the probability weight distribution process, is the key vector The transpose of is the key vector Dimension.

8. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 4, characterized in that: In step 3.2, the feature vector The calculation method is as follows: in, is the linear mapping matrix in the linear transformation parameters that expands the vector dimension by four times, is the bias matrix in the linear transformation parameters that expands the vector dimension by four times, is the linear mapping matrix in the linear transformation parameters that reduces the vector dimension by four times, is the bias matrix in the linear transformation parameters that reduces the vector dimension by a factor of four, is a non-linear activation function.

9. The method for detecting text stance based on an efficient fine-tuned open source large language model according to claim 1, characterized in that: In step 1, the text that needs stance detection is and targets that require stance detection Automatically fill in the prompt word to form a prompt sentence .