Multi-task esg classification method and model for chinese financial text

Abstract

The application discloses a multi-task ESG classification method and model for Chinese financial text, aiming at solving the technical problems of poor pertinence and insufficient accuracy of the existing text classification. The application first performs common latent learning representation on the text according to the word vector as the input, outputs the common latent learning representation, and then inputs the common latent learning representation into the text classification and sentiment word detection tasks for the self-attention mechanism after the full connection layer. Meanwhile, the text is segmented according to the period in the text classification task, sentence vectors are generated, the feature vectors of the label and the last round of the sentiment word detection task are combined, and the combination is input into the Transformer and the linear layer, and the combination is output for text classification by combining the self-attention mechanism. In the sentiment word detection task, the output of the self-attention mechanism and the feature vectors generated in the text classification task are used for sentiment word classification. The two tasks are jointly trained, and the sentence is segmented, so that the problems of long Chinese financial text and difficult classification of emotional extreme words are solved.

Description

Technical Field

[0001] This invention relates to the field of text classification technology, specifically to a multi-task ESG classification method and model for Chinese financial texts. Background Technology

[0002] In the financial sector, there is a wealth of textual information about companies, such as reports, financial statements, and news. Investors often want to analyze this information to gain useful insights into the company's past and present performance, as well as potential future trends, to aid investment decisions. Analyzing the implicit sentiment within the text and categorizing it into sentiment types is a common analytical method. ESG (Environmental, Social, and Governance) evaluation of corporate information is a scientific analytical approach proposed by researchers.

[0003] However, engaging experts to analyze and classify massive amounts of data would be extremely costly in terms of manpower and resources. While pre-trained language models have achieved near-human annotation capabilities in text classification, some models are now used for financial text classification. However, existing pre-trained models still have limitations. For example, they are insufficient for handling long texts, which are often the case in financial texts; the sentiment terms used in corporate ESG assessment-related financial texts differ from those used in general domain sentiment classification; and financial texts related to corporate ESG assessments contain numerous financial terms, with different terms having varying weights in terms of sentiment terms during ESG classification. Summary of the Invention

[0004] This invention provides a multi-task ESG classification method and model for Chinese financial texts to solve the technical problems of poor targeting and insufficient accuracy in existing text classification.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] Design a multi-task ESG classification method for Chinese financial texts, including the following steps:

[0007] S1: Create a text training set in the financial field, where the text contains labels;

[0008] S2: The common latent learning representation part transforms each character in the text into a character vector. The character vectors of all characters form the vector matrix of the text. This vector matrix is ​​input into the feature extraction model to output the feature representation matrix.

[0009] S3: Input the feature representation matrix in S2 into the fully connected layers of the text classification and sentiment word detection parts respectively to obtain new feature representation matrices of the feature vector in the text classification and sentiment word detection spaces. Then, input the new feature representation matrices into the self-attention mechanism to obtain the sentence feature vector formed by the aggregation of the two contextual semantic features of the text classification and sentiment word detection parts.

[0010] S4: The text classification part divides the text into multiple blocks and converts these blocks into text block vectors. In the first round of training, only the text block vectors combined with the text labels are input into the Transformer model. The output of the Transformer is then combined with the sentence feature vectors generated in the S3 text classification part to generate the classification vector P. cla , will P cla The input is fed into the softmax layer for text classification, and the first round of financial text ESG categories is output.

[0011] S5: Sentiment word detection, performed simultaneously with step S4, combines the sentence feature vector from step S3 with the classification vector P from the sentiment word detection part of step S4. cla Combined to form the sentiment word detection vector Z dec and Z dec Input is processed through a softmax layer to segment sentiment words;

[0012] S6: In training rounds after the first round, the text classification part in S4 becomes the sentiment word detection vector Z generated in the previous training round in step S5. dec The text block vectors and text labels are input into the Transformer model and combined with the sentence feature vectors generated by the S3 text classification part. The input is then fed into the softmax layer for text classification, outputting a new round of financial text ESG categories.

[0013] Furthermore, in step S1, the feature extraction model is a BiLSTM model, and the hidden layer output of the BiLSTM model is used as the feature representation matrix to better capture longer-distance dependencies and bidirectional semantic dependencies.

[0014] Furthermore, in step S4, the text is divided into multiple blocks using periods as delimiters.

[0015] Furthermore, in step S1, the creation of the text training set includes: crawling Chinese financial news, financial reports, and annual reports from multiple companies on the Internet and annotating them with six categories: EP, EN, SP, SN, GP, and GN as tags.

[0016] Furthermore, it also includes step S7: joint training, in which steps S2-S6 are repeated for all texts in the training set in S1, and the overall loss is obtained by multiplying the losses of the text classification part and the sentiment word detection part by their respective weights and summing them.

[0017] This invention also designs a multi-task ESG classification model for Chinese financial text, including:

[0018] The Common Latent Learning Representation Module is used to transform text into a vector matrix consisting of individual character vectors, and then extract the features of the vector matrix to output a feature representation matrix;

[0019] The text classification module includes a text segmentation layer, a first fully connected layer, a first self-attention mechanism unit, a Transformer layer, and a first softmax layer. The text segmentation layer divides the text into multiple blocks and generates text block vectors. The first fully connected layer obtains a new feature representation matrix of the feature vector matrix in text classification. The first self-attention mechanism unit obtains the new feature representation matrix and outputs a sentence feature vector aggregated from the contextual semantic features of text classification. The Transformer layer receives the aforementioned text block vectors, text labels, and the sentiment word detection vector Z generated in the previous training round. dec Output classification vector P cla ;

[0020] The sentiment word detection module includes a second fully connected layer, a second self-attention mechanism unit, and a second softmax layer. The second fully connected layer is used to obtain a new feature representation matrix of the feature vector in sentiment word detection. The second self-attention mechanism unit is used to obtain the new feature representation matrix and output a sentence feature vector formed by aggregating the contextual semantic features of sentiment word detection. The second softmax layer is used to receive the sentence feature vector and P from the text classification module. cla Generate sentiment word detection vector Z dec .

[0021] Furthermore, the aforementioned ESG classification model also includes a joint training module, which is used to calculate the loss function of the ESG classification model. This module multiplies the losses of the sentiment word detection module and the text classification module by their respective weights and sums them to obtain the total loss function.

[0022] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0023] 1. In the text classification part of this invention, a text is segmented by periods, and the features of each sentence are extracted into a sentence vector. The sentence vectors are used as the basic units to form a vector representation of the text. This text vector is then combined with the feature vector obtained by BiLSTM for classification, enabling the model to better learn the overall text characteristics and contextual features of long texts. Furthermore, based on this, the text classification and sentiment word detection tasks are combined, and the joint training method enables the model to achieve better results for the characteristics of financial texts.

[0024] 2. This invention targets the classification of Chinese financial texts, enabling sentiment analysis of relevant financial texts with relatively low resource and time costs. Furthermore, as a pre-trained model, this method avoids training with a large amount of annotated data and feature design based on professional domain knowledge, allowing the model to flexibly adjust and learn according to its own text features.

[0025] 3. This invention can quickly, accurately, and with minimal resource requirements complete the ESG classification of financial texts. Investors can use the ESG performance of companies in these texts to assess their investment behavior and their contributions to promoting sustainable economic development and fulfilling social responsibilities, thus providing a reference and guidance for their investment decisions. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the ESG classification model structure of the present invention.

[0027] Figure 2 This is a distribution diagram of the dataset used in this invention.

[0028] Figure 3 Table 1 shows the comparative experimental results of different models in this invention.

[0029] Figure 4 Table 2 shows the comparative experimental results of different models in this invention.

[0030] Figure 5 Table 3 shows the comparative experimental results of different models in this invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.

[0032] Example 1: A multi-task ESG classification method for Chinese financial texts, see [link to example]. Figure 1 It consists of four parts: common latent representation learning, sentiment word detection, text classification, and joint learning.

[0033] 1) Common latent learning representation part: See Figure 1 The section within box 101 transforms each character in the text into a character vector, and these character vectors are combined to form a vector matrix representing the entire text. This vector matrix is ​​then used as input to the BiLSTM model, outputting a feature representation matrix H. D The BiLSTM model here is an existing bidirectional LSTM model. LSTM transmits information useful for computation in subsequent time steps by forgetting information in the cell state and remembering new information, while discarding useless information. Using the hidden layer output of the BiLSTM model as the feature representation matrix can better capture longer-distance dependencies and bidirectional semantic dependencies.

[0034] 2) Sentiment word detection section: See Figure 1 The section within box 102 represents a subtask, primarily used to assist in the text classification task. The Z-factor generated by this subtask... dec This part is incorporated into text classification tasks, allowing text classification tasks to utilize sentiment word features from these tasks. The specific operation of the sentiment word detection part is as follows: the output H of BiLSTM is... D As input, the vectors are first fed into a fully connected layer to obtain a new feature representation matrix of the feature vectors in the sentiment word detection space. This new feature representation matrix is ​​then fed into a self-attention mechanism to obtain sentence features aggregated from contextual semantic features. Self-attention is a common mechanism in neural networks that allows each part of the feature matrix to focus on important parts of the overall text matrix, thus enabling the model to learn contextual semantic features. Finally, the vectors output by the self-attention mechanism are combined with the P vectors from the text classification task. cla Partially combined to form Z dec and Z dec The input is processed by a softmax layer to segment sentiment words. Sentiment words are ultimately divided into six aspects of ESG: P, N, namely EP, EN, SP, SN, GP, GN. B and I represent the beginning and composition of words, and O represents non-sentiment words, for a total of 13 categories.

[0035] 3) Text classification: This is the main task of this method. An article is divided into N blocks (seg1...segN) using periods as delimiters. These blocks are then converted into vectors, allowing the model to input and process longer texts and learn sentence features. Figure 1 The bounding box 103 in the image is then used to combine these vectors with the text labels and the Z vectors generated from the sentiment word detection task in the previous training round. dec Partially input into the Transformer. Add Z. decBy leveraging sentiment word features from the subtask and adding text labels, the model's performance can be enhanced. The Transformer model employs multi-head self-attention mechanisms and residual networks, which largely solve the long-term dependency problem when processing long texts.

[0036] The process of text classification is the same as that of the sub-task; see [link / reference]. Figure 1 The box 104 in the image represents the output H of the BiLSTM. D As input, the data is first fed into a fully connected layer. Then, the new feature representation matrix is ​​fed into a self-attention mechanism to obtain sentence features. These sentence features are then combined with the output of the Transformer above to generate P. cla , will P cla The input is fed into the softmax layer for text classification. Meanwhile, P... cla This information will also be input into the sub-task to assist in its classification. The text classification task yielded a total of 6 ESG categories on P and N.

[0037] 4) Joint Training: The parameters for learning a common latent representation using BiLSTM for the primary and secondary tasks are shared. That is, the output of BiLSTM serves as the input for both the primary and secondary tasks. The overall loss of the model is calculated by adding the losses from the primary and secondary tasks, multiplying by weights beforehand. These weights represent the proportion of each task's contribution to the overall model learning and optimization process, aiming to achieve optimal performance during the model's learning and optimization.

[0038] This invention pertains to pre-trained language models, comprising four parts: common latent learning representation, sentiment word detection, text classification, and joint learning. The pre-training method avoids cumbersome feature design and domain knowledge; by training on a small number of annotated texts, the text can be mapped to a low-dimensional continuous feature vector, which is then automatically classified.

[0039] This invention utilizes Chinese financial news, financial reports, and annual reports of 26 companies crawled from the internet, annotated in six categories, and then uses this as a training set for training. See [link to relevant documentation]. Figure 2 Therefore, the features learned by the model are applicable to Chinese text, and specifically to texts in the financial field. This paper utilizes sentiment word detection and recognition tasks. By training the model on sentiment word recognition, it can learn the characteristics of sentiment words in financial texts that differ from those in general domain texts, as well as the different proportions of sentiment words corresponding to different terms in ESG classification.

[0040] Moreover, the model of this invention classifies financial texts into six categories, both positive and negative, under ESG. This makes the task more complex and the classification granularity finer, which helps investors further analyze company performance and corporate contributions to promoting sustainable economic development and fulfilling social responsibilities.

[0041] This invention uses a BiLSTM network to learn the common latent representation of the corpus. The BiLSTM network is composed of bidirectional LSTMs, which, like LSTMs, can handle long financial texts as input, and compared to LSTMs, can better capture bidirectional semantic dependencies. By mapping Chinese text to character vectors at the character level, it avoids the inaccurate segmentation of long phrase boundaries by word segmentation tools.

[0042] In the text classification module, each text is divided by a period to learn its features and form sentence vectors. These vector features are then combined with the output of the BiLSTM network, which helps the model learn the overall features of the text and can process more text information when dealing with very long texts.

[0043] The self-attention mechanism is used in text classification and sentiment word detection tasks. This mechanism allows the input units to interact with each other and then finds which part of the input it should pay more attention to in the context, so that the contextual semantic features are aggregated into sentence features. When processing the features of long texts, it can learn the contextual features better.

[0044] Finally, this invention combines common latent representation, sentiment word detection, and text classification into a joint learning process, enabling the model to handle long text inputs. Furthermore, it allows the sentiment word detection task within the text and the overall sentiment analysis task to complement each other. Figures 3 to 5 The comparative experiments show that the present invention can achieve better classification results.

[0045] The present invention has been described in detail above with reference to the accompanying drawings and embodiments. However, those skilled in the art will understand that, without departing from the spirit of the present invention, various specific parameters in the above embodiments can be changed to form multiple specific embodiments, all of which are common variations of the present invention, and will not be described in detail here.

Claims

1. A multi-task ESG classification method for Chinese financial texts, characterized in that, Includes the following steps: S1: Create a text training set in the financial field, where the text contains labels; S2: The common latent learning representation part transforms each character in the text into a character vector. The character vectors of all characters form the vector matrix of the text. This vector matrix is ​​input into the feature extraction model to output the feature representation matrix. S3: Input the feature representation matrix in S2 into the fully connected layers of the text classification part and the sentiment word detection part respectively to obtain the new feature representation matrix of the feature vector matrix in text classification and sentiment word detection. Then, input the new feature representation matrix into the self-attention mechanism of the text classification part and the sentiment word detection part respectively to obtain the sentence feature vector formed by the aggregation of the two context semantic features of the text classification part and the sentiment word detection part. S4: The text classification part divides the text into multiple blocks and converts these blocks into text block vectors. In the first round of training, only the text block vectors combined with the text labels are input into the Transformer model. The output of this model is then combined with the sentence feature vectors generated in S3 to generate the classification vector P. cla , will P cla The input is fed into the softmax layer for text classification, and the first round of financial text ESG categories is output. S5: Sentiment word detection, performed simultaneously with step S4, combines the sentence feature vector from step S3 with the classification vector P from the sentiment word detection part of step S4. cla Combined to form the sentiment word detection vector Z dec and Z dec Input is processed through a softmax layer to segment sentiment words; S6: In subsequent training rounds after the first round, the text classification part in S4 becomes the sentiment word detection vector Z generated in the previous training round in step S5. dec The text block vectors and text labels are input into the Transformer model and combined with the sentence feature vectors generated by S3. The input is then fed into the softmax layer for text classification, outputting a new round of financial text ESG categories.

2. The multi-task ESG classification method for Chinese financial text according to claim 1, characterized in that, In step S1, the feature extraction model is a BiLSTM model, and the hidden layer output of the BiLSTM model is used as the feature representation matrix to better capture long-distance dependencies and bidirectional semantic dependencies.

3. The multi-task ESG classification method for Chinese financial texts according to claim 1, characterized in that, In step S4, the text is divided into multiple blocks using periods as delimiters.

4. The multi-task ESG classification method for Chinese financial text according to claim 1, characterized in that, In step S1, the creation of the text training set includes: crawling Chinese financial news, financial reports, and annual reports from multiple companies on the Internet and annotating them with six categories: EP, EN, SP, SN, GP, and GN as tags.

5. The multi-task ESG classification method for Chinese financial text according to claim 1, characterized in that, It also includes step S7: joint training, in which steps S2-S6 are repeated for all texts in the training set in S1, and the overall loss is obtained by multiplying the losses of the text classification part and the sentiment word detection part by their respective weights and summing them.

6. A multi-task ESG classification device for Chinese financial text, used to implement the multi-task ESG classification method for Chinese financial text according to any one of claims 1-5, characterized in that, include: The Common Latent Learning Representation Module is used to transform text into a vector matrix consisting of individual character vectors, and then extract the features of the vector matrix to output a feature representation matrix; The text classification module includes a text segmentation layer, a first fully connected layer, a first self-attention mechanism unit, a Transformer layer, and a first softmax layer. The text segmentation layer divides the text into multiple blocks and generates text block vectors. The first fully connected layer obtains a new feature representation matrix of the feature vector matrix in text classification. The first self-attention mechanism unit obtains the new feature representation matrix and outputs a sentence feature vector aggregated from the contextual semantic features of text classification. The Transformer layer receives the aforementioned text block vectors, text labels, and the sentiment word detection vector Z generated in the previous training round. dec And output the classification vector P cla ; The sentiment word detection module includes a second fully connected layer, a second self-attention mechanism unit, and a second softmax layer. The second fully connected layer is used to obtain a new feature representation matrix of the feature vector in sentiment word detection. The second self-attention mechanism unit is used to obtain the new feature representation matrix and output a sentence feature vector formed by aggregating the contextual semantic features of sentiment word detection. The second softmax layer is used to receive the sentence feature vector and P from the text classification module. cla Generate sentiment word detection vector Z dec .

7. The multi-task ESG classification device for Chinese financial text according to claim 6, characterized in that: Also includes: The joint training module is used to calculate the loss function of the ESG classification model. It multiplies the losses of the sentiment word detection module and the text classification module by their respective weights and sums them to obtain the total loss function.

Images