A hierarchical attention-based method for analyzing interbank funding sentiment.

By constructing a three-level sentiment dictionary and a BERT-BiLSTM model, combined with word-level and sentence-level attention modules, the problem of existing models being unable to accurately identify interbank funding sentiment was solved, achieving more efficient interbank funding sentiment analysis and improving recognition accuracy and generalization ability.

CN118069841BActive Publication Date: 2026-11-10FUDAN UNIVERSITY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410164560.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-05
Publication Date
2026-11-10
Estimated Expiration
2044-02-05

AI Technical Summary

Technical Problem

Existing sentiment classification models cannot accurately identify and classify the interbank funding sentiment expressed in financial news, especially due to a lack of relevant knowledge in the interbank market context, resulting in insufficient recognition capabilities.

Method used

We construct a sentiment analysis method for interbank funding based on hierarchical attention. Through a three-level sentiment dictionary, a BERT-BiLSTM basic classification model, and word-level and sentence-level attention modules, we perform sentence segmentation, encoding, splicing, and sentiment analysis of financial news texts. We use the sub-word representation of the BERT model to guide word-level attention and optimize the model through semi-supervised self-training with pseudo-labels.

Benefits of technology

It improves the model's ability to identify interbank liquidity sentiment, reduces reliance on labeled data, alleviates manual costs, enhances the model's generalization ability, and significantly improves the accuracy of extracting interbank liquidity sentiment from financial news texts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118069841B_ABST
    Figure CN118069841B_ABST
Patent Text Reader

Abstract

This invention provides a hierarchical attention-based method for interbank liquidity sentiment analysis, characterized by the following steps: Step S1, constructing a three-level sentiment dictionary based on multiple existing dictionaries and existing financial news texts; Step S2, constructing and training an improved classification model based on the three-level sentiment dictionary, existing financial news texts, and an existing BERT-BiLSTM basic classification model; Step S3, inputting the financial news texts into the improved classification model to obtain category probability distributions; Step S4, obtaining the interbank liquidity sentiment corresponding to the financial news texts based on the category probability distributions. In summary, this method can extract more accurate interbank liquidity sentiment from financial news texts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a text sentiment analysis method, specifically a sentiment analysis method for interbank funding based on hierarchical attention. Background Technology

[0002] The interbank market is an important component of my country's economic construction and development, playing a crucial role in the financial system.

[0003] Interbank liquidity sentiment refers to the tightness or looseness of liquidity in the interbank market, and is described using positive, neutral, and negative sentiment to indicate whether interbank liquidity is loose, balanced, or tight.

[0004] Some studies use financial news to predict interbank liquidity, for example, by segmenting financial news into words, calculating word vectors, and using machine learning models such as support vector machines, gradient boosting decision trees, and XGBoost for classification.

[0005] However, existing research often uses basic models, such as traditional machine learning models and simple deep learning models. With the rapid development of artificial intelligence and deep learning, compared to traditional models, deep learning technologies such as natural language processing offer an efficient and rapid way to extract semantic information from financial news texts and analyze the underlying sentiment. Because interbank liquidity sentiment exists within the context of the interbank market, it possesses certain unique characteristics compared to general sentiment. General sentiment classification models often lack relevant knowledge of the interbank market and may fail to accurately identify interbank liquidity sentiment information.

[0006] In conclusion, existing sentiment classification models cannot accurately identify and classify the interbank liquidity sentiment expressed in financial news. Summary of the Invention

[0007] This invention is made to solve the above-mentioned problems, and aims to provide a method for interbank funding sentiment analysis based on hierarchical attention.

[0008] This invention provides a hierarchical attention-based interbank funding sentiment analysis method for extracting corresponding interbank funding sentiment from financial news texts. It includes the following steps: Step S1, constructing a three-level sentiment dictionary based on existing dictionaries and existing financial news texts; Step S2, constructing and training an improved classification model based on the three-level sentiment dictionary, existing financial news texts, and an existing BERT-BiLSTM basic classification model; Step S3, inputting the financial news text into the improved classification model to obtain a category probability distribution; Step S4, obtaining the interbank funding sentiment corresponding to the financial news text based on the category probability distribution. The improved classification model includes: a sentence segmentation module for segmenting the text into sentences; and an encoding module containing a cue encoder and a continuous template for encoding the continuous template and respectively... The process involves concatenating individual sentences to obtain concatenated sentences; a BERT module, containing the BERT model, extracts word representations from each concatenated sentence; a data storage module stores a three-level sentiment lexicon; a word-level attention module performs word-level attention processing on each concatenated sentence based on the three-level sentiment lexicon and all word representations of that sentence to obtain a sentence-level representation; a BiLSTM module, containing the BiLSTM model, performs bidirectional context capture on each sentence-level representation to obtain the corresponding processed sentence-level representation; a sentence-level attention module performs sentence-level attention processing on all processed sentence representations to obtain a document-level representation; and a probability distribution module processes the document-level representation sequentially through a fully connected network and a Softmax function to obtain a category probability distribution, where interbank funding sentiment includes positive, neutral, and negative.

[0009] The interbank funding sentiment analysis method based on hierarchical attention provided by this invention may also have the following features: the three-level sentiment dictionary includes a general sentiment dictionary, a financial sentiment dictionary, and a target task sentiment dictionary. Step S1 includes the following sub-steps: Step S1-1, constructing a general sentiment dictionary by merging and deduplicating multiple existing Chinese sentiment dictionaries; Step S1-2, constructing a financial sentiment dictionary by merging and deduplicating multiple existing Chinese financial sentiment dictionaries; Step S1-3, performing data cleaning and word segmentation on existing financial news texts to obtain a candidate word set; Step S1-4, sequentially screening positive and negative words, manually screening, and supplementing all Chinese sentiment dictionaries and the Chinese financial sentiment dictionary to obtain a positive seed word set and a negative seed word set; Step S1-5, using the SO-PMI algorithm, selecting multiple positive words from the candidate word set in conjunction with the positive seed word set; Step S1-6, using the SO-PMI algorithm, selecting multiple negative words from the candidate word set in conjunction with the negative seed word set; Step S1-7, manually screening and supplementing all positive and negative words to obtain the target task sentiment dictionary.

[0010] The interbank funding sentiment analysis method based on hierarchical attention provided in this invention may also have the following feature: In word-level attention processing, the attention score α of the i-th sub-word... i The calculation expression is: In the formula w i It is the sub-word representation of the i-th sub-word, m is the number of sub-words in the sentence, and w [MASK] To represent the sub-words corresponding to the [MASK] special marker in the continuous template of the concatenated sentence, w k For the sub-word representation corresponding to the k-th sub-word, based on the attention score α in the three-level sentiment lexicon... i Make corrections to obtain the corrected attention score. The calculation expression is: In the formula is the average attention score of all sub-words in the sentence, and v is the weight assigned to the words corresponding to the sub-words according to the three-level sentiment dictionary. When the words corresponding to the sub-words appear in the three-level sentiment dictionary, the dictionary weight of the word in the dictionary with higher priority is selected as the weight. The priority of the three-level sentiment dictionary is ordered from low to high as follows: general sentiment dictionary, financial sentiment dictionary, and target task sentiment dictionary.

[0011] The interbank funding sentiment analysis method based on hierarchical attention provided in this invention may also have the following features: the training process of the BERT model includes the following steps: Step T1, constructing a target task keyword dictionary containing multiple keywords based on existing financial domain dictionaries and existing financial news texts; Step T2, combining the target task keyword dictionary, constructing two positive samples and two negative samples for each sentence in the existing financial news texts; Step T3, constructing a training set using the positive and negative samples corresponding to all sentences; Step T4, selecting a sample from the training set and inputting it into the BERT model, combined with random masking... The MLM strategy yields processed [CLS] special tags and multiple processed [MASK] special tags; in step T5, the processed [CLS] special tags are binary classified through the semantic matching output layer to calculate the semantic matching task loss; in step T6, all processed [MASK] special tags are passed through the MLM output layer to obtain prediction results, and the MLM task loss is calculated based on the prediction results; in step T7, the total loss is calculated based on the semantic matching task loss and the MLM task loss to update the BERT model; in step T8, steps T4 to T7 are repeated until the preset conditions are met, thus obtaining the trained BERT model.

[0012] The interbank funding sentiment analysis method based on hierarchical attention provided in this invention may also have the following features: the training process of the improved classification model includes the following steps: Step R1, constructing a labeled dataset and an unlabeled dataset based on financial news text. The labeled dataset includes multiple labeled samples, and the unlabeled dataset includes multiple unlabeled samples. The labeled and unlabeled samples belong to c classes; Step R2, initializing class centers for all classes based on the labeled samples; Step R3, training the improved classification model using the labeled dataset; Step R4, predicting each unlabeled sample based on the improved classification model to obtain the basic pseudo-labels corresponding to each unlabeled sample; Step R5, combining the class centers, performing sharpening, correction, and averaging operations on each basic pseudo-label in sequence. Step R6: Dynamically update the class centers based on the pseudo-labels to obtain updated class centers; Step R7: Set the confidence threshold for each class; Step R8: Use all unlabeled samples with a confidence score greater than the corresponding confidence threshold and all labeled samples as the training set; Step R9: Train the improved classification model based on the training set and update the model parameters of the improved classification model; Step R10: Update each confidence threshold based on the confidence score of all unlabeled samples; Step R11: Repeat steps R8 to R10 until the first training termination condition is met; Step R12: Update each confidence threshold based on the sampling rate; Step R13: Repeat steps R4 to R12 until the second training termination condition is met, thus obtaining the trained improved classification model.

[0013] The interbank funding sentiment analysis method based on hierarchical attention provided by this invention may also have the following feature: wherein, in step R2, the initialization expression is: In the formula n k x is the number of labeled samples belonging to the k-th class. i Let C be the representation of the i-th labeled sample. k Let l be the class center of the k-th class, and l be the total number of labels in the label sample in the label dataset.

[0014] The interbank funding sentiment analysis method based on hierarchical attention provided by this invention may also have the following feature: wherein, in step R5, the expression for the sharpening operation is: In the formula p i,k The basic pseudo-label p of the i-th unlabeled sample before the sharpening operation i The probability of class k, p i,k ' is the basic pseudo-label p of the i-th unlabeled sample after the sharpening operation. i The probability of class k, where c is the total number of classes and T is the temperature parameter, is expressed as follows: In the formula p is the basic pseudo-label after the correction operation corresponding to the i-th unlabeled sample. i Let λ be the basic pseudo-label before the correction operation for the i-th unlabeled sample. d The weights of the distance vector, dist(x) i Let C be the distance from the i-th unlabeled sample to each of the class centers C. The expression for the averaging operation is: In the formula These are the basic pseudo-labels after averaging in the t-th epoch. β represents the basic pseudo-labels before the averaging operation in the t-th epoch, and β is the weighting coefficient.

[0015] The interbank funding sentiment analysis method based on hierarchical attention provided by this invention may also have the following feature: wherein the dynamically updated expression in step R6 is: In the formula n k Let C be the number of samples in class k, l be the total number of labeled samples, nl be the total number of unlabeled samples, and C be the number of samples in class k. k The class center of the k-th class, C represents the dynamically updated class center in the t-th epoch. t This represents the class center before dynamic updates in the t-th epoch.

[0016] The interbank funding sentiment analysis method based on hierarchical attention provided by this invention may also have the following feature: In step R9, the model parameters are updated using the total loss, and the expression for calculating the total loss is: L = L x +λ u L u , In the formula, L represents the total loss. x For label loss, L u For unlabeled loss, λ u The weights are the unlabeled loss values, H() is the cross-entropy loss function, and y is the weights. i For the labeled sample x i The corresponding tag, p m (y|x i To improve the classification model, the labeled samples x i The predicted probability distribution For unlabeled sample x i The corresponding pseudo-tag, p m (y|x' i To improve the classification model for unlabeled samples x' i The predicted probability distribution, in step R10, the expression for updating the confidence threshold is: τt (k)=β τ τ t–1 (k)+(1–β τ )conf(k), In the formula τ t (k) is the confidence threshold for category k after updating at the t-th batch, β τ Here, nl represents the EMA weights, nl is the total number of unlabeled samples, and p is the weights. m (y|x i (k) represents the model for unlabeled samples x i The predicted probability for class k, in step R12, is updated using the following expression for the confidence threshold: In the formula τ e (k) is the confidence threshold of category k after the update in the e-th epoch, τ t (k) is the confidence threshold for category k in the last batch of the e-th epoch, n k This represents the number of unlabeled samples whose pseudo-labels belong to category k and whose confidence level is higher than the confidence threshold corresponding to category k.

[0017] The role and effect of invention

[0018] According to the hierarchical attention-based interbank funding sentiment analysis method of the present invention, firstly, it better utilizes the knowledge of the BERT model in the BERT module through a cue-based learning encoding module, and represents w through word representation. [MASK] First, guided word-level attention effectively mitigates the impact of the label mapper design quality on the model's classification performance, reducing manual costs. Second, by constructing a three-level sentiment dictionary, external sentiment knowledge is introduced into the word-level attention module, enhancing the model's ability to identify interbank funding sentiment. Third, semi-supervised self-training based on pseudo-labels trains the improved classification model, reducing dependence on label data. Dynamic class center correction and adaptive adjustment of confidence thresholds optimize and filter pseudo-labels, effectively reducing the negative impact of erroneous pseudo-labels. Fourth, by constructing a target task keyword dictionary, a keyword-based semantic matching binary classification task is built using keyword replacement and back translation. This task shares underlying parameters with the MLM task, enabling joint adaptive pre-training of the BERT model. Compared to single-task training, joint training allows the BERT model to learn better feature representations, while explicitly and implicitly achieving data augmentation, mitigating overfitting, and improving the model's generalization ability. Therefore, the hierarchical attention-based interbank funding sentiment analysis method of this invention can extract more accurate interbank funding sentiment from financial news texts. Attached Figure Description

[0019] Figure 1This is a flowchart illustrating the interbank funding sentiment analysis method based on hierarchical attention in an embodiment of the present invention.

[0020] Figure 2 This is a block diagram of the improved classification model in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the process of performing joint adaptive pre-training on the BERT model in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram illustrating the principle of joint adaptive pre-training of the BERT model in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram illustrating the process of semi-supervised self-training of the improved classification model in an embodiment of the present invention. Detailed Implementation

[0024] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the interbank funding sentiment analysis method based on hierarchical attention of this invention.

[0025] This embodiment provides a hierarchical attention-based interbank funding sentiment analysis method, which is used to extract corresponding interbank funding sentiment from financial news texts, wherein interbank funding sentiment includes positive, neutral and negative.

[0026] Figure 1 This is a flowchart illustrating the interbank funding sentiment analysis method based on hierarchical attention in an embodiment of the present invention.

[0027] like Figure 1 As shown, the interbank funding sentiment analysis method based on hierarchical attention in this embodiment includes the following steps:

[0028] Step S1: Construct a three-level sentiment dictionary based on the existing multiple dictionaries. The three-level sentiment dictionary includes a general sentiment dictionary, a financial sentiment dictionary, and a target task sentiment dictionary.

[0029] Step S1 includes the following sub-steps:

[0030] Step S1-1: By merging and deduplicating multiple existing Chinese sentiment dictionaries, a general sentiment dictionary is constructed. In this embodiment, the existing HowNet dictionary, the Dalian University of Technology DUTIR sentiment dictionary, and the Tsinghua University Li Jun Chinese Praise and Criticism Dictionary are merged, and duplicate words are extracted to obtain the general sentiment dictionary.

[0031] Step S1-2 involves merging and deduplicating existing Chinese financial sentiment dictionaries to construct a new financial sentiment dictionary.

[0032] Steps S1-3 involve cleaning and segmenting the existing financial news text to obtain a set of candidate words.

[0033] Steps S1-4 involve sequentially filtering positive and negative words, manually filtering, and supplementing all Chinese sentiment dictionaries and Chinese financial sentiment dictionaries to obtain a set of positive seed words and a set of negative seed words.

[0034] In this embodiment, the 200 most frequent positive words and 200 most frequent negative words are first selected from all Chinese sentiment dictionaries and Chinese financial sentiment dictionaries. Then, they are manually selected and supplemented to obtain multiple positive seed words as a positive seed word set and multiple negative seed words as a negative seed word set.

[0035] Steps S1-5: Based on the SO-PMI algorithm, multiple positive words are obtained by filtering from the candidate word set in combination with the positive seed word set.

[0036] Steps S1-6: Based on the SO-PMI algorithm, multiple negative words are selected from the candidate word set by combining the negative seed word set.

[0037] In this embodiment, the PMI value of each candidate word in the candidate word set is calculated relative to the positive seed word set and the negative seed word set. The difference between the PMI value of the positive seed word set and the PMI value of the negative seed word set corresponding to the candidate word is the SO-PMI value of the candidate word. Then, the candidate words with the largest absolute value of SO-PMI value are selected as positive words and negative words. In this embodiment, there are 4000 positive words and 4000 negative words.

[0038] Steps S1-7 involve manually filtering and supplementing all positive and negative words to obtain the target task sentiment dictionary.

[0039] Step S2: Based on the three-level sentiment dictionary, existing financial news texts, and the existing BERT-BiLSTM basic classification model, construct and train an improved classification model.

[0040] Figure 2 This is a block diagram of the improved classification model in an embodiment of the present invention.

[0041] like Figure 2 As shown, the improved classification model 100 in this embodiment includes a sentence segmentation module 10, an encoding module 20, a BERT module 30, a data storage module 40, a word-level attention module 50, a BiLSTM module 60, a sentence-level attention module 70, and a probability distribution module 80.

[0042] Sentence segmentation module 10 is used to segment financial news texts in chapter form into sentences.

[0043] The encoding module 20 includes a prompt encoder and a continuous template, used to encode the continuous template and concatenate it with each sentence to obtain the concatenated sentence corresponding to each sentence. In this embodiment, the prompt encoder consists of a BiLSTM model and two layers of MLP, which can establish connections within the template, i.e., between each prompt subword.

[0044] In this embodiment, the continuous template is a template carrying prompt information. By searching for continuous templates within the continuous embedding space of the pre-trained language model, i.e., the BERT model, and updating the model using these continuous templates as learnable parameters, the overhead of manually finding the optimal template is reduced. This allows the prompt information to be utilized by the pre-trained language model to the maximum extent, better leveraging the model's knowledge.

[0045] BERT module 30 contains the BERT model, which is used to extract the word representations of each word from each concatenated sentence.

[0046] Figure 3 This is a schematic diagram of the process of performing joint adaptive pre-training of the BERT model in an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram illustrating the principle of joint adaptive pre-training of the BERT model in an embodiment of the present invention.

[0048] like Figure 3 , Figure 4 As shown, the joint adaptive pre-training of the BERT model includes a keyword-based semantic matching binary classification task and an MLM (Masked Language Model) task, specifically including the following steps:

[0049] Step T1: Construct a target task keyword dictionary containing multiple keywords based on existing financial field dictionaries and existing financial news texts.

[0050] In this embodiment, the Sogou Finance Financial Glossary, which contains 11,379 common financial terms, is first selected as the financial dictionary. Then, the Top 5 keywords are extracted from each news article in the existing financial news text using TF-IDF. After merging and deduplicating, they are used as a candidate keyword set, totaling 8,955 words. Subsequently, the candidate keyword set is merged with the basic keyword set, namely the Sogou Finance Financial Glossary, and after deduplication, 19,022 words are obtained. Finally, manual screening and supplementation are performed to obtain the final target keyword dictionary, which contains 19,129 keywords.

[0051] Step T2 involves constructing two positive samples and two negative samples for each sentence in the existing financial news text, based on the target task keyword dictionary.

[0052] In this embodiment, the two positive samples are the first positive sample and the second positive sample. The first positive sample is constructed using a keyword replacement method, and the second positive sample is constructed using a reverse translation method. The two negative samples are two sentences randomly selected from existing financial news text that do not contain the original sentence.

[0053] The keyword replacement method is as follows: An existing financial news text is used as the target task corpus to train a Word2Vec model. The trained Word2Vec model can obtain the vector representation of each word. By calculating the cosine similarity of the word vectors, a keyword is replaced with its closest similarity based on the target task keyword dictionary. For each sentence, 10% of the keywords are randomly replaced. If the keywords in a sentence are less than 10% of the total, ordinary words are randomly selected for replacement, thus obtaining the first positive sample for that sentence.

[0054] The reverse translation method involves using the existing Opus-MT machine translation model to translate the sentence in reverse, selecting English as the intermediate language. Because the translation between Chinese and English is generally effective and does not excessively disrupt the semantics of the original sentence, it yields a better second positive sample.

[0055] In this embodiment, both positive and negative samples are sentence pairs. The first positive sample consists of the original sentence and a sentence generated by the keyword replacement method. The second positive sample consists of the original sentence and a sentence generated by the reverse translation method. Both negative samples consist of the original sentence and another sentence from the existing financial news text.

[0056] Step T3: Construct a training set using the positive and negative samples corresponding to all sentences.

[0057] Step T4: Select a sample from the training set, either a positive or negative sample, where Sentence A is the original sentence and Sentence B is another sentence corresponding to the original sentence. Input the sample into the BERT model and combine it with the random masking MLM strategy to obtain the processed [CLS] special tags and multiple processed [MASK] special tags.

[0058] The specific method of the random masking MLM strategy in this embodiment is as follows: A portion of words are randomly selected from the input training sentence. These words typically account for 15% of the total number of words, and are replaced using the [MASK] special tag. The selected words will be replaced with the [MASK] special tag with an 80% probability, replaced with random words with a 10% probability, and remain unchanged with a 10% probability.

[0059] In step T5, the processed [CLS] special tags are binary classified through the semantic matching output layer, i.e., the SM output layer, and the semantic matching task loss, i.e., SM loss, is calculated.

[0060] Step T6: All processed [MASK] special tags are passed through the MLM output layer to obtain prediction results, and the MLM task loss, i.e., MLM loss, is calculated based on the prediction results. In this embodiment, both the semantic matching output layer and the MLM output layer are implemented using a multilayer perceptron.

[0061] Step T7: Calculate the total loss based on the semantic matching task loss and the MLM task loss, and update the BERT model.

[0062] Step T8: Repeat steps T4 to T7 until the preset conditions are met, and you will get the trained BERT model.

[0063] In this embodiment, the BERT model is jointly adaptively pre-trained by synchronously executing the MLM task (MLM task loss) and the keyword-based semantic matching task (semantic matching task loss), enabling the BERT model to learn better representations and achieving data augmentation effects both explicitly and implicitly.

[0064] The data storage module 40 stores a three-level emotion dictionary.

[0065] The word-level attention module 50 is used to perform word-level attention processing on each concatenated sentence based on the three-level sentiment lexicon and the representation of all sub-words of the sentence, so as to obtain the sentence-level representation of the sentence.

[0066] Traditional cue learning methods use a label mapper to transform the BERT model's predicted probability distribution of the [MASK] special tag onto the vocabulary into a classification result. This approach requires designing labels for each category in a specific task, and the quality of the label design directly affects the final classification performance. Furthermore, the coverage of the labels is limited; they must be included in the model's vocabulary, which has certain limitations. Therefore, in this embodiment, instead of using a label mapper, we directly utilize the hidden layer representation of the [MASK] special tag output by the BERT model, i.e., the sub-word representation w. [MASK] Replace the original query vector q word This approach guides word-level attention, thereby mitigating the impact of the label mapper's quality on the model's classification performance, reducing manual costs, and more closely integrating cue learning with the classification model. In word-level attention processing, the attention score α of the i-th sub-word using the above method... i The calculation expression is:

[0067]

[0068] In the formula w i It is the sub-word representation of the i-th sub-word, m is the number of sub-words in the sentence, and w [MASK] To represent the sub-words corresponding to the [MASK] special marker in the continuous template of the concatenated sentence, w k This represents the subword corresponding to the k-th subword.

[0069] Based on the three-level emotion dictionary, attention score α i Make corrections to obtain the corrected attention score. The calculation expression is:

[0070]

[0071] In the formula is the average attention score of all sub-words in the sentence, v is the weight assigned to the words corresponding to the sub-words according to the three-level sentiment lexicon, and softmax is the softmax function used for normalization.

[0072] When a word corresponding to a sub-word appears in the third-level sentiment dictionary, the dictionary weight of that word in the dictionary with the highest priority is selected as the weight. The priority of the third-level sentiment dictionaries, from lowest to highest, is: general sentiment dictionary, financial sentiment dictionary, and target task sentiment dictionary. In this embodiment, the positions of all sub-words corresponding to words that do not appear in the third-level sentiment dictionary are set to zero. The dictionary weight is a learnable parameter, which can have better weight values ​​than manually assigned fixed dictionary weights.

[0073] The BiLSTM module 60 contains a BiLSTM model used to perform bidirectional context capture on each sentence-level representation to obtain the corresponding processed sentence-level representation.

[0074] The sentence-level attention module 70 is used to perform sentence-level attention processing on all processed sentence representations to obtain discourse-level representations.

[0075] The probability distribution module 80 is used to process the document-level representation sequentially through a fully connected network and a Softmax function to obtain the category probability distribution.

[0076] Figure 5 This is a schematic diagram illustrating the process of semi-supervised self-training of the improved classification model in an embodiment of the present invention.

[0077] like Figure 5 As shown, the training process of the improved classification model 100 includes the following steps:

[0078] Step R1: Construct a labeled dataset and an unlabeled dataset based on the financial news text. The labeled dataset includes multiple labeled samples, and the unlabeled dataset includes multiple unlabeled samples. The labeled and unlabeled samples belong to c classes.

[0079] Step R2: Initialize the class centers of all classes based on the label samples.

[0080] In step R2, the initialization expression is:

[0081]

[0082]

[0083] In the formula n k x is the number of labeled samples belonging to the k-th class. i Let C be the representation of the i-th labeled sample. k Let l be the class center of the k-th class, and l be the total number of labels in the label sample in the label dataset.

[0084] Step R3: Train the improved classification model 100 using the labeled dataset.

[0085] Step R4: Based on the improved classification model 100, predict each unlabeled sample to obtain the basic pseudo-label corresponding to each unlabeled sample.

[0086] Step R5: Based on the class center, sharpen, correct, and average the basic pseudo-labels in sequence to obtain the corresponding pseudo-labels.

[0087] In step R5, the expression for the sharpening operation is:

[0088]

[0089] In the formula p i,k The basic pseudo-label p of the i-th unlabeled sample before the sharpening operation i The probability of class k, p i,k ' is the basic pseudo-label p of the i-th unlabeled sample after the sharpening operation. i The probability of class k is given by c, where c is the total number of classes and T is the temperature parameter. The closer the temperature parameter T is to zero, the closer the basic pseudo-label is to one-hot encoding.

[0090] The expression for the correction operation is:

[0091]

[0092] In the formula p is the basic pseudo-label after the correction operation corresponding to the i-th unlabeled sample.i Let λ be the basic pseudo-label before the correction operation for the i-th unlabeled sample. d The weights of the distance vector, dist(x) i ,C) represents the distance from the i-th unlabeled sample to each of the class centers C.

[0093] The expression for averaging is:

[0094]

[0095] In the formula These are the basic pseudo-labels after averaging in the t-th epoch. Let β be the basic pseudo-label before averaging in the t-th epoch, and β be the weighting coefficient. In this embodiment, the basic pseudo-label of the unlabeled sample after averaging is the pseudo-label corresponding to that unlabeled sample. In this embodiment, the EMA method is used for averaging through the weighting coefficient β, thereby reducing the impact of model misprediction.

[0096] Step R6: Dynamically update each class center based on the pseudo-labels to obtain the updated class centers.

[0097] The expression for dynamic update, i.e., dynamic class center correction, in step R6 is:

[0098]

[0099]

[0100]

[0101] In the formula n k Let be the number of samples in class k, which includes not only labeled samples belonging to class k but also unlabeled samples with pseudo-labels belonging to class k. Let l be the total number of labeled samples, nl be the total number of unlabeled samples, and C be the total number of samples in class k. k The class center of the k-th class, C represents the dynamically updated class center in the t-th epoch. t Let be the class centers before dynamic updates in the t-th epoch, and β be the EMA weight, used to control the update intensity; the larger β is, the smaller the update intensity. In this embodiment, the EMA method used is the exponential moving average method, which is a time-weighted average, with higher weights for more recent data. Updating the class centers using this EMA method makes the values ​​of the class centers more stable and reliable.

[0102] Step R7: Set the confidence threshold for each class.

[0103] Step R8: Use all unlabeled samples with confidence scores greater than the corresponding confidence threshold in the corresponding class and all labeled samples as the training set.

[0104] Step R9: Train the improved classification model 100 based on the training set and update the model parameters of the improved classification model 100.

[0105] In step R9, the model parameters are updated using the total loss, and the expression for calculating the total loss is as follows:

[0106] L = L x +λ u L u ,

[0107]

[0108]

[0109] In the formula, L represents the total loss. x For label loss, L u For unlabeled loss, λ u The weights are the unlabeled loss values, H() is the cross-entropy loss function, and y is the weights. i For the labeled sample x i The corresponding tag is pm(y|x) i To improve the classification model, 100 pairs of labeled samples x i The predicted probability distribution For unlabeled sample x i The corresponding pseudo-tag, p m (y|x' i To improve the classification model, 100 pairs of unlabeled samples x' i The predicted probability distribution.

[0110] Step R10: Update each confidence threshold based on the confidence of all unlabeled samples.

[0111] In step R10, the expression for updating the confidence threshold is:

[0112] τ t (k)=β τ τ t–1 (k)+(1–β τ )conf(k),

[0113]

[0114] In the formula τ t (k) is the confidence threshold for category k after updating at the t-th batch, β τ Here, nl represents the EMA weights, nl is the total number of unlabeled samples, and p is the weights.m (y|x i (k) represents the model for unlabeled samples x i The predicted probability in the k-th class.

[0115] Step R11: Repeat steps R8 to R10 until the first training termination condition is met.

[0116] Step R12: Update each confidence threshold according to the sampling rate.

[0117] In step R12, the expression for updating the confidence threshold is:

[0118]

[0119]

[0120] In the formula τ e (k) is the confidence threshold of category k after the update in the e-th epoch, τ t (k) is the confidence threshold for category k in the last batch of the e-th epoch, n k This represents the number of unlabeled samples whose pseudo-labels belong to category k and whose confidence level is higher than the confidence threshold corresponding to category k.

[0121] Determining the confidence threshold in conventional training is challenging. A high confidence threshold may lead to the discarding of samples with correct pseudo-labels; a low threshold may introduce excessive pseudo-label noise during training, failing to provide effective filtering. Therefore, existing techniques require extensive experimentation to select a reasonable confidence threshold, which is time-consuming and labor-intensive. In this embodiment, by executing steps R10 and R12, the confidence threshold is adaptively adjusted by category during training based on the sampling rate (samp(k) and average confidence (conf(k)) of unlabeled samples in each category. The confidence threshold automatically changes and adjusts during training, thus better preserving correct pseudo-labels and eliminating incorrect pseudo-labels, thereby improving model training performance.

[0122] Step R13, repeat steps R4 to R12 until the second training termination condition is met, then the trained improved classification model 100 is obtained.

[0123] Step S3: Input the financial news text into the improved classification model to obtain the category probability distribution.

[0124] Step S4: Based on the category probability distribution, obtain the interbank liquidity sentiment corresponding to the financial news text.

[0125] This embodiment compares the hierarchical attention-based interbank funding sentiment analysis method, BABA-CT-LDict-SSL, with several existing supervised methods: TextCNN, BiLSTM, BiLSTM-Att, HAN, BERT, BERT-Att, BABA, BABA-CT, BABA-CT-Lex, and BABA-CT-LDict, as well as several existing semi-supervised methods: BABA-CT-LDict-NT, BABA-CT-LDict-FT, and BABA-CT-LDict-FreeMatch, on an existing financial news dataset. The analysis uses four metrics: accuracy, precision, recall, and F1 score. The specific results are shown in the table below.

[0126] method accuracy Accuracy Recall rate F1 score TextCNN 0.7096 0.5478 0.5379 0.5367 BiLSTM 0.6003 0.4649 0.4199 0.4184 BiLSTM-Att 0.6738 0.5508 0.5294 0.5326 HAN 0.7014 0.6079 0.5208 0.5130 BERT 0.7439 0.5638 0.5694 0.5593 BERT-Att 0.7178 0.5865 0.5617 0.5671 BABA 0.7406 0.6753 0.5982 0.6100 BABA-CT 0.7553 0.6545 0.6110 0.6233 BABA-CT-Lex 0.7537 0.6746 0.6136 0.6165 BABA-CT-LDict 0.7520 0.6577 0.6460 0.6394 BABA-CT-LDict-NT 0.7683 0.6758 0.6402 0.6483

[0127] In the table above, the first column represents each method, and the second to fifth columns represent the accuracy, precision, recall, and F1 score for each method, respectively. It is evident that compared to BERT and BERT-Att methods, which directly treat the input news text as sentences, the BABA-CT-LDict-SSL method leverages the discourse characteristics of news text, extracting information at both the word and sentence levels, resulting in significant improvements across all four evaluation metrics. Compared to the HAN method, which also employs a hierarchical attention mechanism, BABA, using both BERT and BiLSTM models, significantly improves model performance, with an F1 score improvement of approximately 10%. Building upon the BABA model, the improved classification model BABA-CT, which integrates cue learning, utilizes continuous templates to more effectively leverage the knowledge of the pre-trained language model, resulting in improved classification performance. Based on the BABA-CT model, the improved classification model BABA-CT-LDict, which integrates a three-level sentiment lexicon, enhances the model's ability to recognize interbank funding sentiment by introducing additional sentiment knowledge at the word-level attention layer. Compared to the BABA-CT model, it shows significant improvements in precision, recall, and F1 score, with the F1 score reaching 0.6394. Under semi-supervised conditions, the BABA-CT-LDict-SSL method, based on sampling rate and average confidence, uses an adaptive confidence threshold to classify and filter pseudo-labels, achieving better results than other pseudo-label filtering methods. It further improves across all four performance metrics, with precision reaching 0.7781 and an F1 score of 0.6569.

[0128] This embodiment also analyzes the effectiveness of the joint adaptive pre-training method, namely the MLM+SM method, comparing it with existing MLM methods, MLM+NSP methods, and methods without pre-training (No method). The analysis also uses four metrics: accuracy, precision, recall, and F1 score. The specific analysis results are shown in the table below:

[0129] method accuracy Accuracy Recall rate F1 score No 0.7781 0.6845 0.6456 0.6569 MLM 0.7586 0.6650 0.6428 0.6497 SM 0.7928 0.7392 0.6382 0.6596 MLM+NSP 0.7716 0.6816 0.6588 0.6645 MLM+SM 0.7716 0.7141 0.6538 0.6738

[0130] In the table above, the first column represents each method, and the second to fifth columns represent the accuracy, precision, recall, and F1 score for each method, respectively. It can be seen that the keyword-based semantic matching binary classification task (SM method) used in the hierarchical attention-based interbank funding sentiment analysis method of this embodiment has achieved good results. The MLM+SM method, obtained by jointly training SM with the MLM method, can further improve model performance, showing significant improvements in both recall and F1 score.

[0131] The role and effect of the embodiments

[0132] According to the hierarchical attention-based interbank funding sentiment analysis method involved in this embodiment, firstly, by using a prompt-based learning-based encoding module, the knowledge of the BERT model in the BERT module is better utilized, and w is represented by sub-words. [MASK] First, guided word-level attention effectively mitigates the impact of the label mapper design quality on model classification performance, reducing manual costs. Second, by constructing a three-level sentiment dictionary, external sentiment knowledge is introduced into the word-level attention module, enhancing the model's ability to identify interbank funding sentiment. Third, semi-supervised self-training based on pseudo-labels trains the improved classification model, reducing dependence on label data. Dynamic class center correction and adaptive adjustment of confidence thresholds optimize and filter pseudo-labels, effectively reducing the negative impact of erroneous pseudo-labels. Fourth, by constructing a target task keyword dictionary, a keyword-based semantic matching binary classification task is built using keyword replacement and back translation. This task shares underlying parameters with the MLM task, enabling joint adaptive pre-training of the BERT model. Compared to single-task training, joint training allows the BERT model to learn better feature representations, while explicitly and implicitly implementing data augmentation, mitigating overfitting, and improving the model's generalization ability. In summary, this method can extract more accurate interbank funding sentiment from financial news texts.

[0133] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A hierarchical attention-based interbank funding sentiment analysis method for extracting corresponding interbank funding sentiment from financial news texts, characterized in that... Includes the following steps: Step S1: Construct a three-level sentiment dictionary based on multiple existing dictionaries and existing financial news texts; Step S2: Based on the three-level sentiment dictionary, the existing financial news text, and the existing BERT-BiLSTM basic classification model, construct and train an improved classification model. Step S3: Input the financial news text into the improved classification model to obtain the category probability distribution; Step S4: Based on the category probability distribution, obtain the interbank liquidity sentiment corresponding to the financial news text. The improved classification model includes: The sentence segmentation module is used to segment the financial news text in chapter form into multiple sentences. The encoding module includes a prompt encoder and a continuous template, used to encode the continuous template and concatenate it with each sentence to obtain the concatenated sentence corresponding to each sentence; The BERT module contains the BERT model, which is used to extract the word representations corresponding to each word from each of the concatenated sentences. The data storage module stores the aforementioned three-level sentiment dictionary; The word-level attention module is used to perform word-level attention processing on each of the concatenated sentences based on the three-level sentiment dictionary and the sub-word representations of the sentence, so as to obtain the sentence-level representation of the sentence; The BiLSTM module contains a BiLSTM model, which is used to perform bidirectional context capture on each of the sentence-level representations to obtain the corresponding processed sentence-level representations. The sentence-level attention module is used to perform sentence-level attention processing on all the processed sentence representations to obtain the document-level representation; The probability distribution module is used to process the document-level representation sequentially through a fully connected network and a Softmax function to obtain the category probability distribution. The interbank funding sentiment is categorized as positive, neutral, and negative. The training process of the improved classification model includes the following steps: Step R1: Construct a labeled dataset and an unlabeled dataset based on the financial news text. The labeled dataset includes multiple labeled samples, and the unlabeled dataset includes multiple unlabeled samples. The labeled samples and the unlabeled samples belong to c classes. Step R2: Initialize the class centers of all the classes based on the label samples; Step R3: Train the improved classification model using the labeled dataset; Step R4: Based on the improved classification model, predict each of the unlabeled samples to obtain the basic pseudo-labels corresponding to each of the unlabeled samples; Step R5: Based on the class center, perform sharpening, correction and averaging operations on each of the basic pseudo-labels in sequence to obtain the corresponding pseudo-labels; Step R6: Dynamically update each class center according to the pseudo-label to obtain the updated class centers; Step R7: Set the confidence threshold for each of the aforementioned classes; Step R8: Use all unlabeled samples with a confidence level greater than the corresponding confidence threshold for the class and all labeled samples as the training set; Step R9: Train the improved classification model based on the training set and update the model parameters of the improved classification model; Step R10: Update each of the confidence thresholds based on the confidence of all the unlabeled samples; Step R11: Repeat steps R8 to R10 until the first training termination condition is met; Step R12: Update each of the confidence thresholds according to the sampling rate; Step R13: Repeat steps R4 to R12 until the second training termination condition is met, and the trained improved classification model is obtained.

2. The interbank funding sentiment analysis method based on hierarchical attention as described in claim 1, Its features are: The three-level sentiment dictionary includes a general sentiment dictionary, a financial sentiment dictionary, and a target task sentiment dictionary. Step S1 includes the following sub-steps: Step S1-1: By merging and deduplicating multiple existing Chinese sentiment dictionaries, the general sentiment dictionary is constructed. Step S1-2: The financial sentiment dictionary is constructed by merging and deduplicating multiple existing Chinese financial sentiment dictionaries. Steps S1-3 involve data cleaning and word segmentation of the existing financial news text to obtain a candidate word set; Steps S1-4: For all the Chinese sentiment dictionaries and the Chinese financial sentiment dictionaries, positive and negative words are sequentially screened, manually screened, and supplemented to obtain a positive seed word set and a negative seed word set; Steps S1-5: According to the SO-PMI algorithm, multiple positive words are selected from the candidate word set by combining the positive seed word set; Steps S1-6: According to the SO-PMI algorithm, multiple negative words are selected from the candidate word set by combining the negative seed word set; Steps S1-7 involve manually screening and supplementing all the positive and negative words to obtain the target task sentiment dictionary.

3. The interbank funding sentiment analysis method based on hierarchical attention according to claim 1, characterized in that: in, In the aforementioned word-level attention processing, the first Attention score of each of the sub-words The calculation expression is: , In the formula It is the sub-word representation of the i-th sub-word. The number of sub-words mentioned in the sentence. This refers to the sub-words corresponding to the [MASK] special marker in the continuous template of the concatenated sentence. For the first Sub-word representations corresponding to each sub-word Based on the attention score of the three-level emotion dictionary Make corrections to obtain the corrected attention score. The calculation expression is: , In the formula This is the average of the attention scores corresponding to all the aforementioned subwords in the sentence. The weights assigned to the words corresponding to the sub-words based on the three-level sentiment dictionary. When the word corresponding to the sub-word appears in the third-level sentiment dictionary, the dictionary weight of the word in the dictionary with the highest priority is selected as the weight. The priority of the three-level sentiment dictionaries, from lowest to highest, is as follows: general sentiment dictionary, financial sentiment dictionary, and target task sentiment dictionary.

4. The interbank funding sentiment analysis method based on hierarchical attention as described in claim 1, Its features are: The training process of the BERT model includes the following steps: Step T1: Construct a target task keyword dictionary containing multiple keywords based on existing financial field dictionaries and the existing financial news text; Step T2: Combining the target task keyword dictionary, construct two positive samples and two negative samples for each sentence in the existing financial news text; Step T3: Construct a training set using the positive and negative samples corresponding to all the sentences as samples; Step T4: Select one sample from the training set and input it into the BERT model. Combine the random masking MLM strategy to obtain the processed [CLS] special label and multiple processed [MASK] special labels. Step T5: The processed [CLS] special tags are binary classified through the semantic matching output layer to calculate the semantic matching task loss; Step T6: Obtain prediction results by passing all the processed [MASK] special tags through the MLM output layer, and calculate the MLM task loss based on the prediction results; Step T7: Calculate the total loss based on the semantic matching task loss and the MLM task loss, and update the BERT model accordingly; Step T8: Repeat steps T4 to T7 until the preset conditions are met, and the trained BERT model is obtained.

5. The interbank funding sentiment analysis method based on hierarchical attention according to claim 1, characterized in that: in, In step R2, the initialization expression is: , , In the formula For belonging to the first The number of labeled samples for each class. For the first Representation of a labeled sample, For the first The class center of each class, This represents the total number of labels in the labeled dataset.

6. The interbank funding sentiment analysis method based on hierarchical attention according to claim 1, characterized in that: in, In step R5, the expression for the sharpening operation is: , In the formula The first step before sharpening operation The basic pseudo-label of an unlabeled sample In the The probability of a class The first after the sharpening operation The basic pseudo-label of an unlabeled sample In the The probability of a class The total number of all classes. For temperature parameters, The expression for the correction operation is: , In the formula For the first The basic pseudo-labels after correction for each unlabeled sample For the first The basic pseudo-labels before the correction operation for each unlabeled sample. The weights of the distance vector. For the first Each unlabeled sample was assigned to all class centers. distance, The expression for the averaging operation is: , In the formula For the first The basic pseudo-labels after averaging across epochs. For the first Basic pseudo-labels before averaging in each epoch These are the weighting coefficients.

7. The interbank funding sentiment analysis method based on hierarchical attention according to claim 1, characterized in that: in, The expression for dynamic update in step R6 is: , , , In the formula For the first The number of samples in the class The total number of the labeled samples. The total number of the unlabeled samples. For the first Class center of the class, For the first The class center is dynamically updated in each epoch. For the first The class center before dynamic updates in each epoch.

8. The interbank funding sentiment analysis method based on hierarchical attention according to claim 1, characterized in that: in, In step R9, the model parameters are updated using the total loss, which is calculated as follows: , , , In the formula For the total loss, For label loss, For unlabeled loss, The weights are for unlabeled loss. Let cross-entropy be the loss function. For label samples The corresponding tags To improve the classification model for labeled samples The predicted probability distribution Unlabeled samples The corresponding pseudo-tags, To improve the classification model for unlabeled samples The predicted probability distribution In step R10, the expression for updating the confidence threshold is: , , In the formula For the first Each batch of categories Updated confidence threshold For EMA weights, The total number of the unlabeled samples. For the model to test unlabeled samples In the Predicted probabilities on class In step R12, the expression for updating the confidence threshold is: , , In the formula For the first Updated categories in each epoch The confidence threshold For the first The categories in the last batch of each epoch The confidence threshold The pseudo-tag belongs to the category And the confidence level is higher than that of the class. The number of unlabeled samples corresponding to the confidence threshold.