Text sentiment classification method, electronic equipment and storage medium

Abstract

The invention provides a text sentiment classification method, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-analyzed financial text; inputting a financial text to be analyzed into the FinBERT model to obtain an embedded representation form; inputting the to-be-analyzed financial text into the self-supervised learning model, learning the semantic role relationship of sentences, and inputting the to-be-analyzed financial text into the supervised learning model to obtain global features and local features; wherein the supervised learning model comprises a first module for capturing local features of a text, a second module for capturing global features of the text and an attention mechanism layer; and determining a text sentiment classification result corresponding to the financial text to be analyzed according to output results of the self-supervised learning model and the supervised learning model. According to the method and the device, a mode of combining the FinBERT model, the self-supervised learning model and the supervised learning model can be adopted, so that the limitation of a single type of model on financial text information identification and extraction is avoided, and the accuracy of an emotion classification result is improved.

Description

Technical Field

[0001] The present application relates to the technical field of text analysis, and in particular to a text sentiment classification method, an electronic device, and a storage medium. Background Art

[0002] Text sentiment analysis is the process of analyzing, processing, summarizing and reasoning subjective texts with emotional colors. The Internet has generated a large amount of valuable comments on people, events, products, etc., which are participated by users. These comments express people's various emotional colors and emotional tendencies, such as joy, anger, sadness, happiness, criticism, praise, etc. Based on this, potential users can understand the public opinion on a certain event or product by browsing these subjective comments.

[0003] In the financial field, there is a large amount of information such as financial news, company announcements and investor comments, which often contain emotional attitudes towards related events or listed companies. Investors are often not completely rational, and their decisions are affected by personal emotions. For example, positive emotions towards a listed company may prompt investors to increase their holdings of the company's stock, while fear of market turmoil may cause them to sell the stock prematurely. Behavioral finance believes that financial emotions play an important role in investment decisions, which further affects market trends. For investors and market analysts, quickly and accurately capturing these emotional signals is of great significance to investment decisions.

[0004] Traditional text sentiment analysis is usually based on dictionary analysis methods and machine learning-based analysis methods. The dictionary-based analysis method determines the sentiment polarity of the text by matching the words extracted from the text with the words in the sentiment dictionary. This method has some problems, such as the quality of the sentiment dictionary directly affects the classification results, and it is impossible to perform sentiment analysis based on contextual semantics. The machine learning-based analysis method uses traditional machine learning models to extract and understand sentiment information from large amounts of data. This method usually relies on manually designed features, and requires too much manual intervention when selecting features. The semantics of financial texts are usually complex and diverse, and traditional methods find it difficult to capture the deeper emotional meanings in the text. In addition, the emotional expression of the text often depends on the specific context. Traditional methods lack a deep understanding of contextual information, which limits the accuracy of sentiment analysis.

[0005] With the rapid development of deep learning technology, especially the introduction of the Transformer architecture, sentiment analysis methods based on pre-trained language models have gradually become the forefront of research. Their bidirectional encoding capabilities can better understand the semantic relationship of the context, especially pre-trained models represented by Bidirectional Encoder Representations from Transformers (BERT). By pre-training on large-scale corpora, they can capture rich language features and achieve higher sentiment analysis accuracy through fine-tuning on specific domain tasks, providing new possibilities for financial sentiment analysis.

[0006] In the process of implementing the embodiments of the present application, it is found that there are at least the following technical problems in the related technology:

[0007] These general models have poor accuracy in identifying and extracting specific financial sentiment information. Summary of the invention

[0008] The embodiments of the present application provide a text sentiment classification method, an electronic device, and a storage medium to solve the problem of poor accuracy in identifying and extracting specific financial sentiment information by general models.

[0009] In a first aspect, an embodiment of the present application provides a text sentiment classification method, comprising:

[0010] Obtain the financial text to be analyzed;

[0011] Inputting the financial text to be analyzed into the FinBERT model to obtain an embedded representation;

[0012] Inputting the financial text to be analyzed into a self-supervised learning model to learn the semantic role relationship of sentences, and inputting the financial text to be analyzed into a supervised learning model to obtain global features and local features; wherein the supervised learning model includes a first module for capturing local features of the text, a second module for capturing global features of the text, and an attention mechanism layer;

[0013] According to the output results of the self-supervised learning model and the supervised learning model, a text sentiment classification result corresponding to the financial text to be analyzed is determined.

[0014] In a possible implementation, the types of text sentiment classification include positive, neutral and negative.

[0015] In a possible implementation, before obtaining the financial text to be analyzed, the method further includes:

[0016] Obtain comprehensive financial texts from different channels; the different channels include: stock forum comments, financial news headlines and listed company announcements;

[0017] Determining the annotation information of the sentiment classification of each text in the comprehensive financial text;

[0018] Dividing the annotation information and the comprehensive financial text according to the sentiment classification of each text to obtain a financial text annotation data set;

[0019] The comprehensive model constructed by the FinBERT model, the self-supervised learning model and the supervised learning model is trained according to the financial text annotation dataset.

[0020] In a possible implementation, determining the labeling information of each text sentiment classification in the comprehensive financial text includes:

[0021] Obtaining annotation results of two or more annotation channels corresponding to the comprehensive financial text;

[0022] The annotation results of each annotation channel are cross-checked and verified to obtain the annotation information.

[0023] In a possible implementation, inputting the financial text to be analyzed into the FinBERT model to obtain an embedded representation includes:

[0024] The financial text to be analyzed is input into the FinBERT model, and the vector of the last hidden layer and the CLS vector of each hidden layer are extracted.

[0025] In a possible implementation, when the global feature and the local feature are input into a supervised learning model, the method includes:

[0026] Inputting the local features into the first module, and inputting the global features into the second module;

[0027] The output results of the first module and the second module are input into the attention mechanism layer for feature fusion.

[0028] In a possible implementation, the FinBERT model includes: a text embedding layer and a Transformer encoder;

[0029] The text embedding layer classifies and vectorizes the financial text; the vectorization results include TokenEmbedding, Positional Embedding and Segment Embedding;

[0030] The Transformer encoder includes multiple layers of encoders, and the output data of each layer of encoders is used as the input data of the next layer of encoders; each layer of encoders includes: an attention mechanism layer, a combination layer and a linear layer; the input data of each layer of encoders is sequentially input into the attention mechanism layer, the combination layer, the linear layer and the combination layer to obtain the output data.

[0031] In a possible implementation, the first module is constructed based on an improved Text-CNN; the second module is constructed based on an improved BiLSTM-Attention; and the attention mechanism layer of the supervised learning model adopts a multi-head attention mechanism.

[0032] In a possible implementation, the self-supervised learning module includes: an LTP module and a transformer encoder;

[0033] The step of inputting the financial text to be analyzed into a self-supervised learning model to learn the semantic role relationship of sentences includes:

[0034] Identify predicates and arguments in the financial text to be analyzed by the LTP module, and assign corresponding semantic roles to the arguments;

[0035] According to the semantic role assignment results, a transformer encoder is used to predict the masked semantic roles in the semantic role space.

[0036] In a possible implementation, the improved Text-CNN includes four convolution kernels;

[0037] Among them, the first convolution kernel is connected to the K-Max-pooling pooling method;

[0038] The second convolution kernel is connected to the Max-pooling module;

[0039] The third convolution kernel is connected to the Max-pooling module and the average pooling module respectively;

[0040] The fourth convolution kernel is a 1×1 convolution kernel, which takes the last hidden layer feature output by the FinBERT model as input;

[0041] The splicing result of the pooling output data corresponding to the first convolution kernel, the second convolution kernel and the third convolution kernel and the convolution result corresponding to the fourth convolution kernel are fused as the output result of the improved Text-CNN.

[0042] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0043] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any possible implementation of the first aspect.

[0044] The embodiment of the present application provides a text sentiment classification method, an electronic device and a storage medium. By obtaining the financial text to be analyzed, the text is input into the FinBERT model, and the embedded representation form is obtained by using the model to enhance the semantic understanding ability of the model for the financial text. The financial text to be analyzed is input into the self-supervised learning model and the supervised learning model respectively, and the accuracy of the sentiment analysis of the financial text is improved by combining the self-supervised learning model and the supervised learning model. Among them, the self-supervised learning model autonomously learns context information and semantic relationships, and the supervised learning model includes a first module for capturing local features of the text, a second module for capturing global features of the text, and an attention mechanism layer, and adopts a dual-channel result structure to fully capture text information. According to the output results of the self-supervised learning model and the supervised learning model, the text sentiment classification result corresponding to the financial text to be analyzed is determined to improve the accuracy of the sentiment analysis. The embodiment of the present application adopts a combination of the FinBERT model, the self-supervised learning model and the supervised learning model to avoid the limitations of a single type of model in the identification and extraction of financial text information, thereby improving the accuracy of the sentiment classification results and providing reference information for dynamic monitoring and investment decisions of the financial market. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 This is a flowchart of an implementation method of a text sentiment classification method provided in an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of data set length distribution provided by a specific embodiment of the present application;

[0048] Figure 3It is a schematic diagram of the structure of the FinBERT model provided in one embodiment of the present application;

[0049] Figure 4 is a schematic diagram of the structure of a FinBERT encoder provided in an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the structure of an improved Text-CNN provided in an embodiment of the present application;

[0051] Figure 6 It is a schematic diagram of the overall structure of a text sentiment classification model provided in an embodiment of the present application;

[0052] Figure 7 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0054] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present application described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0055] Unless otherwise specified, the term "plurality" means two or more. The character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B. The term "and / or" is a description of the association relationship between objects, indicating that there can be three relationships. For example, A and / or B means: A or B, or, A and B.

[0056] The words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include the plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or including (comprising) refer to the existence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or these groups. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the process, method or device including the elements.

[0057] In this application, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts may refer to the description of the method part.

[0058] There are several major problems and deficiencies in the existing Chinese financial text sentiment analysis technology:

[0059] First, the current public datasets for sentiment analysis of financial texts are English datasets. There are relatively few publicly available Chinese financial sentiment analysis datasets, and most of them are concentrated on data from specific platforms, such as stock bar comments. Such datasets are usually accompanied by high noise and have certain inconsistencies in annotation, which may lead to low data quality problems during model training and evaluation, making it difficult to fully reflect the diverse emotional expressions in the financial market, thus affecting the generalization ability of the model and the actual application effect;

[0060] Second, although most pre-trained models such as BERT or RoBERTa can effectively capture contextual relationships and perform well in many natural language processing tasks, these general models may not be sufficient to identify and extract specific financial sentiment information in sentiment analysis in the financial field. Financial texts are highly professional and have unique expressions. When dealing with these specific situations, existing pre-trained models may not be able to fully capture subtle emotional changes, which in turn affects the accuracy of the analysis.

[0061] Third, relying solely on a single model architecture such as Convolutional Neural Networks (CNN) or Bidirectional Long Short-Term Memory (BiLSTM) is unable to simultaneously learn both the global contextual features and the key local features in the text. CNN is good at extracting local information, but may ignore long-distance contextual relationships; and although BiLSTM can capture longer dependencies, it performs poorly when processing local details. Therefore, how to effectively combine the advantages of different models to fully capture the global and local sentiment features in financial texts is still an important challenge facing current technology.

[0062] Fourth, the semantic roles in current financial texts (such as agent, patient, predicate, etc.) often have complex contextual dependencies. Especially when expressing emotional information such as company performance, policy changes, and market reactions, slight differences in these semantic roles may lead to different emotional interpretations. Traditional supervised learning models usually rely on a large amount of labeled data and are difficult to fully capture these delicate semantic relationships.

[0063] This application aims to construct financial text sentiment analysis by integrating the FinBERT model, self-supervised learning model and supervised learning model to improve the accuracy of recognition and extraction of specific financial sentiment information.

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0065] Figure 1 is a flowchart of the implementation of the text sentiment classification method provided by an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0066] S101, obtaining a financial text to be analyzed.

[0067] The execution subjects of each embodiment of the present application can be servers, processors, microprocessors and other devices with data processing functions. In the actual implementation process, the specific implementation method of the execution subject can be selected according to actual needs. This embodiment does not impose any special restrictions on this, as long as it is a device with data processing functions.

[0068] The financial texts to be analyzed may include but are not limited to various finance-related documents, reports, analysis articles, market reviews, transaction records, etc.

[0069] In the process of acquiring financial texts, a variety of technical means can be used.

[0070] Alternatively, financial texts can be obtained from the Internet using crawler technology. In order to enrich the data source of the dataset, financial text data can be crawled from multiple channels such as stock bar comments, financial news headlines, and listed company announcements. These channels can provide financial information from different angles and levels, which helps to build a comprehensive and diversified dataset.

[0071] Among other possible implementation methods, financial texts to be analyzed can also be obtained through manual collection, partner sharing, public data set download, etc. Selection and combination are carried out according to actual conditions and available resources to ensure that high-quality and highly relevant financial text data are obtained.

[0072] S102: Input the financial text to be analyzed into the FinBERT model to obtain an embedded representation.

[0073] S103, inputting the financial text to be analyzed into a self-supervised learning model to learn the semantic role relationship of the sentence, and inputting the financial text to be analyzed into a supervised learning model to obtain global features and local features; wherein the supervised learning model includes a first module for capturing local features of the text, a second module for capturing global features of the text, and an attention mechanism layer.

[0074] In the self-supervised learning module, the Language Technology Platform (LTP) is used to perform semantic role labeling (SRL) on financial texts, mainly identifying the agent, patient, and predicate in the text, thereby obtaining a concise representation of the financial text. Then, one or more semantic roles are randomly replaced with all-zero vectors to achieve random masking of semantic roles. This requires the model to be able to predict the masked semantic roles from the semantic role candidate space, thereby improving the model's ability to understand contextual semantic relationships.

[0075] The process of semantic role labeling involves identifying the predicates and their related arguments in a sentence, and assigning corresponding semantic roles (such as agent, patient, predicate, etc.) to these arguments, thereby constructing the semantic structure of the sentence. Combining semantic role labeling helps the model better understand the contextual information of the text, thereby more accurately analyzing the text sentiment. The goal of the self-supervised learning module is to predict the randomly masked semantic roles in financial texts. The specific operation is to randomly mask a semantic role in the financial text and require the model to identify the masked semantic role from the semantic role candidate space based on the contextual information. In this way, the model can gradually grasp the semantic structure of the sentence. This method not only improves the accuracy of sentiment analysis, but also enhances the robustness of the model, especially when dealing with complex and incomplete financial texts, showing stronger context perception.

[0076] The supervision module adopts a dual-channel architecture that integrates the first and second modules to capture the global and local features of the text. Subsequently, the features extracted by the two channels are fused and sent to the attention mechanism layer for processing. The purpose is to simultaneously learn the global and local features of the text and extract rich feature representations from different angles, thereby improving the accuracy of text sentiment analysis. The loss function of the entire task is composed of the losses of the self-supervision module and the supervision module, and finally the classification of sentiment polarity is achieved through the output layer.

[0077] S104, determining the text sentiment classification result corresponding to the financial text to be analyzed according to the output results of the self-supervised learning model and the supervised learning model.

[0078] By combining self-supervised learning with supervised learning, we can not only extract rich feature representations from different angles, but also enhance the contextual understanding of complex and incomplete financial texts, effectively improve the ability to process complex inputs, and thus improve the classification performance of financial text sentiment analysis.

[0079] In this embodiment, by obtaining the financial text to be analyzed, it is input into the FinBERT model, and the model is used to obtain an embedded representation to enhance the model's ability to understand the semantics of the financial text. The financial text to be analyzed is input into the self-supervised learning model and the supervised learning model respectively, and the accuracy of the sentiment analysis of the financial text is improved by combining the self-supervised learning model and the supervised learning model. Among them, the self-supervised learning model autonomously learns context information and semantic relationships, and the supervised learning model includes a first module that captures local features of the text, a second module that captures global features of the text, and an attention mechanism layer, and adopts a dual-channel result structure to fully capture text information. According to the output results of the self-supervised learning model and the supervised learning model, the text sentiment classification result corresponding to the financial text to be analyzed is determined to improve the accuracy of the sentiment analysis. The embodiment of the present application adopts a combination of the FinBERT model, the self-supervised learning model and the supervised learning model to avoid the limitations of a single type of model in the identification and extraction of financial text information, thereby improving the accuracy of the sentiment classification results and providing reference information for dynamic monitoring and investment decisions in the financial market.

[0080] The above-mentioned embodiments introduce the main process of text sentiment classification. According to different embodiments, the types of text sentiment classification may be different.

[0081] In a possible implementation, the types of text sentiment classification may include positive, neutral, and negative.

[0082] Among them, positive, neutral and negative types indicate that the text reflects the stock trend as rising, stable and falling respectively. This classification method helps investors better understand market sentiment and make more informed investment decisions.

[0083] In other possible implementations, the types of text sentiment classification may be further refined, including highly positive, generally positive, neutral, generally negative, and highly negative.

[0084] This more detailed classification can provide richer information and help investors more accurately grasp the changes in market sentiment. For example, highly positive text may indicate that the market's expectations for a certain stock are very high, while generally negative text may indicate that the market's expectations for a certain stock are relatively low. Through this classification, investors can better understand the nuances of market sentiment and make more precise investment decisions.

[0085] In addition, in order to ensure the accuracy of text sentiment classification results, the model needs to be fully trained before performing sentiment analysis on financial texts.

[0086] In a possible implementation, before obtaining the financial text to be analyzed, the following is further included:

[0087] Obtain comprehensive financial texts from different channels; the different channels include: stock forum comments, financial news headlines and listed company announcements;

[0088] Determine the annotation information of the sentiment classification of each text in the comprehensive financial text;

[0089] The annotation information and comprehensive financial texts are divided according to the sentiment classification of each text to obtain a financial text annotation dataset;

[0090] The comprehensive model constructed by the FinBERT model, self-supervised learning model and supervised learning model is trained based on the financial text annotation dataset.

[0091] Among them, by collecting comprehensive financial texts from multiple channels, rich and diverse information can be obtained, which helps to understand the sentiment and dynamics of the financial market more deeply. Determine the sentiment classification annotation information of each text in these comprehensive financial texts to lay the foundation for subsequent model training. Usually, these annotation information are classified by professional annotators according to the sentiment tendency of the text content, for example, they are classified into positive, negative or neutral. Then, based on the annotation information of these text sentiment classifications, the comprehensive financial texts can be classified to form one or more financial text annotation datasets containing different sentiment classifications. These datasets provide comprehensive and rich training materials for model training.

[0092] Using these financial text annotation datasets, the FinBERT model, the self-supervised learning model, and the comprehensive model constructed by the supervised learning model are trained. In this way, the model can learn the characteristics of different sentiment classifications and show higher accuracy and reliability in the actual financial text sentiment classification task.

[0093] In this embodiment, by integrating comprehensive financial texts from different channels and constructing a data set based on the different sentiment classification annotation information of these texts, and then training the model based on the data set, it can be ensured that the model can more accurately identify and classify the sentiment tendencies in the text when processing specific financial text sentiment classification tasks.

[0094] In a possible implementation, determining the labeling information of each text sentiment classification in the comprehensive financial text includes:

[0095] Obtain the annotation results of two or more annotation channels corresponding to the comprehensive financial text;

[0096] The annotation results of each annotation channel are cross-checked and verified to obtain the annotation information.

[0097] Among them, obtaining the annotation results of two or more annotation channels corresponding to the comprehensive financial text aims to obtain annotation information through multiple different annotation channels to ensure that data can be obtained from multiple angles and sources. These annotation channels can be different annotation teams, different annotation tools or different annotation methods to ensure the diversity and comprehensiveness of the annotation results.

[0098] Secondly, the annotation results of each annotation channel are cross-checked and verified to obtain the final annotation information. This step is to ensure the accuracy and consistency of the annotation results. Through cross-checking and verification, errors and deviations that may occur in the annotation process can be discovered and corrected, thereby improving the reliability of the annotation results. Specifically, the results of different annotation channels can be compared to find out the differences and contradictions, and through further discussion and analysis, a consistent annotation result can be reached.

[0099] In this embodiment, the annotation information is obtained by cross-checking and verifying the annotation results of two or more annotation channels, which can effectively determine the annotation information of each text sentiment classification in the comprehensive financial text, improve the accuracy of the annotation results, thereby improving the accuracy of the model training, and providing a stable foundation for the subsequent text sentiment classification of financial texts.

[0100] The preparation work before the above model training is described with a specific embodiment.

[0101] 1: Use crawler technology to obtain financial texts from the Internet. In order to enrich the data source of the data set, the following three types of financial text data are crawled respectively: ①: Stock bar comments. The stock bar of Eastmoney.com is a platform website for retail investors to communicate, which can directly reflect the emotions of retail investors. This embodiment of the application randomly selected 11,490 comment data from the stock bar forum of Eastmoney; ②: Financial news headlines. Financial news headlines often express the main idea of ​​a piece of news. This embodiment of the application collected 4,802 financial news headlines from the Internet; ③: Listed company announcements, which refer to company-related information announced to the public through designated platforms by listed companies in accordance with the requirements of the China Securities Regulatory Commission. It has high authority. This embodiment of the application collected 16,040 listed company announcements.

[0102] 2: Data cleaning: In order to reduce the impact of noisy data on model performance, the original crawled text needs to be cleaned. First, remove invalid symbols in the text, such as '&', '*', '#', '@', etc. In addition, clean up HTML tags contained in the text, such as paragraph tags " "、Hyperlink tag" "At the same time, URL links such as "www.baidu.com" are removed to reduce interference factors in unstructured data. Finally, to further improve the relevance and accuracy of the data, regular expressions are used to remove content that is not related to financial text, such as telephone numbers.

[0103] 3: Data labeling: In order to ensure data quality and perform cross-check verification, two financial professionals will label the entire data set separately. Each person will label the entire data set independently to ensure data quality and perform cross-check verification. When the labeling results of the two people are inconsistent for the same data, the two labelers will jointly review to complete the final labeling. According to the sentiment tendency of the text, positive labeling is 2, negative labeling is 0, and neutral labeling is 1. The final completed data set sample is shown in Table 1:

[0104] Table 1 Dataset examples

[0105]

[0106] The samples in the data set are counted, where the shortest data length is 4 characters, the longest data length is 150 characters, and the average length of all samples is 61.21 characters. The specific data set length distribution is as follows: Figure 2 shown.

[0107] Among them, the data proportions of the three categories are shown in Table 2:

[0108] Table 2 Proportion of data in three categories

[0109]

[0110]

[0111] The dataset is randomly divided into training set, validation set and test set according to 7:1.5:1.5, see Table 3.

[0112] Table 3 Financial sentiment analysis dataset

[0113]

[0114] The comprehensive model constructed by the FinBERT model, the self-supervised learning model, and the supervised learning model is trained according to the training set, the validation set, and the test set.

[0115] The above introduces the training process of the model. Next, we will explain the details of the model.

[0116] In one possible implementation, the financial text to be analyzed is input into the FinBERT model to obtain global features and local features, including:

[0117] Input the financial text to be analyzed into the FinBERT model, and extract the vector of the last hidden layer and the CLS vector of each hidden layer.

[0118] Among them, FinBERT is a pre-trained model for the financial field based on the BERT architecture. It uses a large amount of financial corpus for training, so it is suitable for processing financial text data. The traditional method of extracting semantic features based on FinBERT generally only extracts the features of the last hidden layer. The embodiment of the present application uses the features of different layers of FinBERT for different feature extraction channels. Specifically, the features of the last hidden layer are extracted for the BiLSTM-Attention channel to capture the temporal dependencies and key sentiment information of the context; the "[CLS]" vector of each hidden layer is extracted for the first module of the supervised learning model to extract local features.

[0119] FinBERT is divided into a text embedding layer and a Transformer encoder layer. The specific structure is as follows Figure 3 As shown in the figure, the preprocessed financial text is input into the Transformer encoder layer after the text embedding layer, and the features of the last hidden layer and the "[CLS]" vector extracted from each hidden layer are output.

[0120] In one possible implementation, the FinBERT model includes: a text embedding layer and a Transformer encoder;

[0121] The text embedding layer classifies and vectorizes financial texts; the vectorization results include TokenEmbedding, Positional Embedding, and Segment Embedding.

[0122] The Transformer encoder includes multiple layers of encoders, and the output data of each layer of encoders is used as the input data of the next layer of encoders; each layer of encoders includes: attention mechanism layer, combination layer and linear layer; the input data of each layer of encoders is sequentially input into the attention mechanism layer, combination layer, linear layer and combination layer to obtain the output data.

[0123] like Figure 3 As shown in the figure, the text embedding layer vectorizes the financial text and consists of three parts: TokenEmbedding, Positional Embedding, and Segment Embedding.

[0124] Token Embedding: This process maps financial text into a low-dimensional dense vector space. A financial text contains s characters. Find the index position of these s characters in the FinBERT vocabulary and then obtain the vector representation of each character. After Token Embedding, a financial text data is converted into a tensor. Where s represents the number of characters contained in the data, and n represents the dimension of each character embedding.

[0125] Positional Embedding: Each character in a text sequence has a position, and position encoding is used to distinguish different positions. After position embedding, we get a tensor

[0126] Segment Embedding: In order to distinguish different parts of the input sequence, a common word embedding is used to distinguish the position of each sequence. After segment embedding, the tensor is obtained

[0127] E=T+P+S

[0128] in, m represents the maximum sentence length, and b represents the data of each batch.

[0129] The multi-layer encoder uses the FinBERT-Base version, which includes a total of 12 layers of Transformer encoders, including attention mechanism layers, regularization layers, linear layers, and Drouput layers. The detailed structure of the encoder is as follows Figure 4 As shown. The specific processing process of the encoder of each layer of FinBERT is as follows:

[0130] 1. After the attention mechanism layer:

[0131] First, the input E is mapped to Q, K, and V through three weight matrices. Among them: W Q , W K , W V is the weight matrix.

[0132] Q=E×W Q

[0133] K=E×W K

[0134] V=E×W V

[0135] Then calculate the attention score:

[0136]

[0137] MultiHead(Q,K,V)=Concat(head1,…,head n )

[0138] where head i =Attention(QW i Q ,KW i K ,VW i V )

[0139] After obtaining the attention score, the dimension is changed so that the output E of the attention mechanism layer A The same dimension as the input.

[0140] 2. After the Dropout layer, residual connection and regularization layer:

[0141] E A = Dropout(E A )

[0142] E N =LayerNorm(E+E A )

[0143] 3. After the linear layer, it passes through the Dropout layer, residual connection and regularization layer again to get the final output:

[0144] E N = Dropout(E N )

[0145] E O =LayerNorm(E A +E N )

[0146] The output of each encoder is used as the input of the next encoder, and the final output is: the "[CLS]" vector of each hidden layer and the features of the last hidden layer Among them: b represents each batch of data, e represents the number of layers containing Transformer encoders in FinBERT, and h represents the size of the hidden layer dimension of FinBERT.

[0147] In a possible implementation, when global features and local features are input into a supervised learning model, it includes:

[0148] Inputting the local features into the first module and the global features into the second module;

[0149] The output results of the first module and the second module are input into the attention mechanism layer for feature fusion.

[0150] Based on the detailed introduction of the FinBERT model in the aforementioned embodiment, local features are input into the first module, and global features are input into the second module, that is, the vector of the last hidden layer of the FinBERT model is extracted and input into the second module, and the CLS vector of each hidden layer is extracted and input into the first module.

[0151] In a possible implementation, the first module is constructed based on the improved Text-CNN; the second module is constructed based on the improved BiLSTM-Attention; and the attention mechanism layer of the supervised learning model adopts a multi-head attention mechanism.

[0152] In this embodiment, BiLSTM is used to capture context information and dependencies in the sequence, and the attention mechanism is used to enhance the focus on key sentiment information, so as to more accurately classify sentiment polarity. The multi-head attention mechanism extracts rich feature representations from different angles, enhances the fusion effect of BiLSTM and Text-CNN output features, and improves the expressiveness and robustness of the model.

[0153] In addition, the output results of the BiLSTM-Attention channel and the improved Text-CNN channel are concatenated and input into the attention mechanism layer. A multi-head attention mechanism is used here to extract rich feature representations from different angles through multiple attention heads. This not only enhances the fusion effect of the BiLSTM and TextCNN output features, but also improves the expressiveness and robustness of the model. The multi-head attention mechanism can focus on different parts of the input data and capture complex patterns and relationships, thereby improving the model's classification performance and ability to handle complex inputs.

[0154] Concatenate the output results of the BiLSTM-Attention channel and the improved Text-CNN channel:

[0155] H=BcontactT

[0156] The output of this layer is obtained through the multi-head attention mechanism. The formula of the multi-head attention mechanism has been explained in the previous article:

[0157] M=Multihead(H)

[0158] After the attention mechanism layer, it is input into the fully connected layer to predict the sentiment polarity category. The formula is as follows:

[0159] S = Softmax(w f M+b f )

[0160] Where: w f is the weight of the fully connected layer, b f is the bias term.

[0161] In one possible implementation, the improved Text-CNN includes four convolution kernels;

[0162] Among them, the pooling method of K-Max-pooling is connected after the first convolution kernel; K-Max-pooling retains the first K maximum values ​​instead of a single maximum value, which can capture more significant features in the input;

[0163] The Max-pooling module is connected after the second convolution kernel to retain the most significant features in each convolution window, so that the model can capture key information more effectively.

[0164] The third convolution kernel is connected to the Max-pooling module and the average pooling module respectively. The maximum pooling retains the most significant features, while the average pooling retains the average information in the entire convolution window. The combination of the two can capture more global information while retaining important features.

[0165] The fourth convolution kernel is a 1×1 convolution kernel, which takes the last hidden layer feature output by the FinBERT model as input;

[0166] The concatenation results of the pooling output data corresponding to the first convolution kernel, the second convolution kernel, and the third convolution kernel and the convolution result corresponding to the fourth convolution kernel are fused as the output result of the improved Text-CNN.

[0167] In this embodiment, 1×1 convolution and mixed pooling strategies are introduced to improve the structure of traditional Text-CNN, effectively retain more semantic features, and solve the problem of feature loss in the convolution process.

[0168] The improved Text-CNN pipeline uses the "[CLS]" vector extracted from each hidden layer of FinBERT as input. The lower layers of FinBERT capture more vocabulary and syntactic information, while the higher layers capture more semantic information. By integrating the CLS vectors of each layer, these different levels of information can be obtained at the same time. Based on the original Text-CNN, 1×1 convolution and hybrid planning strategies are introduced to obtain richer local features. The improved Text-CNN structure is as follows Figure 5 shown.

[0169] Based on the original Text-CNN, an additional 1×1 convolution kernel is introduced. The input of the 1×1 convolution is the feature of the last hidden layer of FinBERT. Through the convolution operation, the high-dimensional features of the last hidden layer are transformed into a new feature representation, which is fused with the output of Text-CNN, thereby enhancing the effect of feature fusion.

[0170] The K-Max-pooling pooling method is used after the first convolution kernel of Text-CNN. The size of the first convolution kernel is the smallest, and the semantic features obtained after the convolution operation are the richest. Using K-Max-pooling retains the first K maximum values ​​instead of a single maximum value, which can capture more significant features in the input. In addition, these K maximum values ​​have a positional relationship, which can retain position-sensitive features.

[0171] The Max-pooling pooling method is used after the second convolution kernel, which can retain the most significant features in each convolution window, so that the model can capture key information more effectively.

[0172] After the third convolution kernel, a combination of maximum pooling and average pooling is used. Maximum pooling retains the most significant features, while average pooling retains the average information within the entire convolution window. The combination of the two can capture more global information while retaining important features.

[0173] The convolution formula is as follows:

[0174] c i =f(ω*x i:i+h-1 )

[0175] Among them, c i is the result of the convolution operation, ω is the convolution kernel, x i:i+h-1 is a sliding window consisting of rows from i to i+h-1, and f is a nonlinear activation function.

[0176] The output results of the three convolution kernels are concatenated and then fused with the result of the 1×1 convolution. The output of the improved Text-CNN channel is obtained. Where c represents the number of convolution kernels used.

[0177] In one possible implementation, the self-supervised learning module includes: an LTP module and a transformer encoder;

[0178] The financial text to be analyzed is input into the self-supervised learning model to learn the semantic role relationship of the sentence, including:

[0179] Identify the predicates and arguments in the financial text to be analyzed through the LTP module, and assign corresponding semantic roles to the arguments;

[0180] According to the semantic role assignment results, a transformer encoder is used to predict the masked semantic roles in the semantic role space.

[0181] In this embodiment, the goal of the self-supervision module is to identify randomly masked semantic roles. First, it labels the semantic roles in the financial text, and then predicts the randomly masked semantic roles to improve the model's ability to understand the contextual semantic relationship.

[0182] The above embodiments respectively describe the supervised learning model and the FinBERT model. The following describes the text sentiment classification model after the combination of the models. The overall structure diagram of the text sentiment classification model is shown in FIG. Figure 6 shown.

[0183] like Figure 6 As shown, the self-supervised learning module uses the LTP platform to label the semantic roles in the financial text. The specific method is: label the semantic roles in each piece of financial text data through the LTP platform. The present application embodiment labels three types of roles: agent (A0), object (A1) and predicate (V). For example: "The actual controller increases the company's shares", the result after semantic role labeling is: actual controller (A0), company shares (A1), increase holdings (V).

[0184] Get the semantic role embedding representation. Use the financial field pre-trained model FinBERT to process each financial text and get the financial text embedding representation {s0,s1,…,s n }, use a0, a1, v to represent the index of A0, A1, V in the embedding representation of the entire text respectively, and finally pool a0, a1, v to obtain the embedding representation of each semantic role:

[0185]

[0186] Compactly represent financial texts and represent each piece of financial text using semantic roles:

[0187]

[0188] Perform self-supervised training:

[0189] The task of self-supervised training is defined as predicting randomly masked semantic roles in financial text. Randomly select a semantic role from the compact representation of financial text Replace e with a vector of all 0s j represents the masking operation, thereby obtaining the masked financial text representation:

[0190]

[0191] E S ′ Input into Transformer to get the query vector Q S :

[0192] Q S = Transformer(E S ′)

[0193] Input the same type of semantic roles in the semantic role candidate space into the Transformer to obtain K S :

[0194]

[0195] Q S and K S The product is performed, and the Softmax function is used to expect the highest matching score at the i-th position. Then the semantic role at the i-th position is the masked semantic role predicted by the model:

[0196] Score = Q S *K S .

[0197] The overall task loss is composed of the self-supervised learning module and the supervised learning module:

[0198]

[0199] Among them: Loss SSL Loss is the self-supervised learning loss. SL is the supervised learning loss, is the weight parameter.

[0200] In this implementation, the supervised learning module uses the different hierarchical features of the FinBERT model, combined with the global semantic features of the last hidden layer and the "[CLS]" local features of each hidden layer, to improve the model's ability to understand the semantics of financial texts. By inputting the last hidden layer of FinBERT and the "[CLS]" vector of each hidden layer into the dual-channel structure, this method effectively combines global semantic features and local features, enabling the model to understand the semantics of financial texts more comprehensively. The dual-channel feature extraction structure adopts a dual-channel structure of BiLSTM-Attention channel and improved Text-CNN channel, which captures text information comprehensively by extracting global semantic features and local features respectively. The improved Text-CNN structure introduces 1×1 convolution kernels and mixed pooling strategies to enhance the ability of local feature extraction, solve the problem of feature loss in traditional Text-CNN, and retain richer semantic information. Self-supervised learning randomly masks semantic roles and requires the model to predict the masked roles. The model autonomously learns contextual information and semantic relationships, improving the accuracy of financial sentiment analysis and the robustness of the model. In summary, the overall model design for improving the accuracy of sentiment analysis of financial texts is achieved by combining the self-supervised learning module and the supervised learning module.

[0201] In this study, the multi-layer features of the FinBERT model are used to fuse the global semantic features of the last layer with the "[CLS]" local features of each layer, thereby enhancing the model's understanding of the semantics of financial text. In supervised learning, the global and local features are successfully integrated by inputting the last hidden state of FinBERT and the "[CLS]" vector of each hidden layer into the dual-channel structure, enabling the model to capture the semantic content of financial text more comprehensively. The dual-channel feature extraction structure uses the BiLSTM-Attention channel and the improved Text-CNN channel to extract global semantic features and local features respectively, and comprehensively capture text information. The improved Text-CNN structure introduces 1×1 convolution kernels and mixed pooling strategies to improve the ability of local feature extraction, solve the loss problem of traditional Text-CNN in feature extraction, and retain richer semantic information. Self-supervised learning enables the model to autonomously learn contextual information and semantic relationships by randomly masking semantic roles and predicting the masked roles, thereby improving the accuracy of financial sentiment analysis and the robustness of the model. The loss of the overall task is composed of the losses of the two modules. In summary, by combining the self-supervised learning module and the supervised learning module, the designed overall model significantly improves the accuracy of sentiment analysis of financial texts.

[0202] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0203] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned various text sentiment classification method embodiments are implemented, such as Figure 1 The steps shown.

[0204] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 72 in the electronic device 7.

[0205] The electronic device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0206] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0207] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0208] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0209] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0210] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0211] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various text sentiment classification method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium.

[0215] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A text sentiment classification method, characterized in that: include: Obtain the financial text to be analyzed; Inputting the financial text to be analyzed into the FinBERT model to obtain an embedded representation; Inputting the financial text to be analyzed into a self-supervised learning model to learn the semantic role relationship of sentences, and inputting the financial text to be analyzed into a supervised learning model to obtain global features and local features; wherein the supervised learning model includes a first module for capturing local features of the text, a second module for capturing global features of the text, and an attention mechanism layer; According to the output results of the self-supervised learning model and the supervised learning model, a text sentiment classification result corresponding to the financial text to be analyzed is determined.

2. The method according to claim 1, characterized in that The types of text sentiment classification include positive, neutral and negative.

3. The method according to claim 2, characterized in that Before obtaining the financial text to be analyzed, the method further includes: Obtain comprehensive financial texts from different channels; the different channels include: stock forum comments, financial news headlines and listed company announcements; Determining the annotation information of the sentiment classification of each text in the comprehensive financial text; Dividing the annotation information and the comprehensive financial text according to the sentiment classification of each text to obtain a financial text annotation data set; The comprehensive model constructed by the FinBERT model, the self-supervised learning model and the supervised learning model is trained according to the financial text annotation dataset.

4. The method according to claim 3, characterized in that: The step of determining the annotation information of the sentiment classification of each text in the comprehensive financial text includes: Obtaining annotation results of two or more annotation channels corresponding to the comprehensive financial text; The annotation results of each annotation channel are cross-checked and verified to obtain the annotation information.

5. The method according to claim 1, characterized in that The step of inputting the financial text to be analyzed into the FinBERT model to obtain an embedded representation includes: The financial text to be analyzed is input into the FinBERT model, and the vector of the last hidden layer and the CLS vector of each hidden layer are extracted.

6. The method according to claim 5, characterized in that When the global features and the local features are input into a supervised learning model, the method includes: Inputting the local features into the first module, and inputting the global features into the second module; The output results of the first module and the second module are input into the attention mechanism layer for feature fusion.

7. The method according to claim 5, characterized in that The FinBERT model includes: a text embedding layer and a Transformer encoder; The text embedding layer classifies and vectorizes the financial text; the vectorization results include TokenEmbedding, Positional Embedding and Segment Embedding; The Transformer encoder includes multiple layers of encoders, and the output data of each layer of encoders is used as the input data of the next layer of encoders; each layer of encoders includes: an attention mechanism layer, a combination layer and a linear layer; the input data of each layer of encoders is sequentially input into the attention mechanism layer, the combination layer, the linear layer and the combination layer to obtain the output data.

8. The method according to claim 1, characterized in that: The self-supervised learning module includes: an LTP module and a transformer encoder; The step of inputting the financial text to be analyzed into a self-supervised learning model to learn the semantic role relationship of sentences includes: Identify predicates and arguments in the financial text to be analyzed by the LTP module, and assign corresponding semantic roles to the arguments; According to the semantic role assignment results, a transformer encoder is used to predict the masked semantic roles in the semantic role space.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method as claimed in any one of claims 1 to 8 are implemented.

Images