A Financial Text Classification Method Based on Graph Comparison Learning

By constructing text graphs and graph neural networks based on graph contrastive learning, the problem of insufficient utilization of unlabeled sample information in financial text classification is solved, and more efficient and accurate text classification results are achieved.

CN117932068BActive Publication Date: 2026-11-17EAST CHINA NORMAL UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410098876.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-24
Publication Date
2026-11-17
Estimated Expiration
2044-01-24

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively utilize unlabeled sample information and external knowledge in financial text classification, and traditional methods are inefficient, with noise impacting model robustness.

Method used

We employ a graph-based contrastive learning approach, which involves constructing a text graph and a graph neural network. We utilize the graph structure for self-supervised learning, integrate external knowledge and data augmentation techniques, and train the graph neural network to capture the semantic and syntactic relationships of the text.

Benefits of technology

It improves the accuracy and efficiency of financial text classification, makes full use of unlabeled sample information, reduces the impact of noise on the model, and enhances the robustness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117932068B_ABST
    Figure CN117932068B_ABST
Patent Text Reader

Abstract

This invention relates to a financial text classification method based on graph contrastive learning, which addresses the problems of numerous categories, insufficient annotations, and a large number of proper nouns in financial text classification tasks by using a graph neural network model. The method includes the following steps: vectorizing the documents; obtaining the relationships between texts by incorporating external knowledge, and constructing a graph of document nodes based on the documents and their relationships; constructing a graph neural network feature propagation model, inputting the text graph into the graph neural network model, and training the model through contrastive learning; and using the trained model to perform classification prediction on a test set. Through these steps, this invention effectively incorporates knowledge from the financial domain, improving the model's understanding and classification performance on unlabeled data. Overall, this invention provides an effective method for financial text classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a financial text classification method based on graph convolutional neural networks and contrastive learning. Background Technology

[0002] Text classification is a crucial research topic in Natural Language Processing (NLP). It aims to automate the processing and analysis of large amounts of text data using machine learning methods. By labeling each text segment according to given classification criteria, it serves as a fundamental method and prerequisite for many other tasks. Text classification has wide applications in sentiment analysis, spam filtering, topic tagging, news classification, and other scenarios. Text representation learning, a vital step in text classification, can be divided into two main categories: engineered features and learned features. Traditional feature engineering methods based on classification models primarily rely on manually crafted features, such as the bag-of-words method, and utilize traditional machine learning methods like Naive Bayes and Support Vector Machines, which are labor-intensive and inefficient. Deep learning methods have been used to address this problem, and can be categorized into sequence-based deep learning models such as convolutional neural networks and recurrent neural networks, and graph-based models such as graph convolutional networks. Compared to sequence-based models, graph-based models have the advantage that graph neural networks can handle complex structured data.

[0003] Graph Neural Networks (GNNs) are deep learning models based on graph structures that can effectively handle complex non-Euclidean data, such as social networks. In recent years, GNNs have seen increasing research and application in text classification tasks. GNNs can capture structured text data, semantic and syntactic relationships within the text, and understand text structure at different levels of abstraction, thereby capturing semantic and contextual information. Furthermore, they can incorporate knowledge specific to financial texts, such as named entities, to help the model understand domain-specific text. By constructing graph structures between documents, GNNs can capture the similarities and relevance between documents, thus improving text classification performance.

[0004] Graph contrastive learning is a self-supervised learning method that utilizes graph-structured data. Its goal is to learn a graph embedding function that maps each node or graph to a low-dimensional vector space, such that nodes with higher similarity are closer together. The first step in graph-based text representation learning is constructing the text graph. Traditional methods generate text graphs on the entire corpus, but this introduces noise that affects the robustness of graph neural network models. Data augmentation of the graph helps mitigate the negative impact of noise on the model. On the other hand, training using self-supervised learning helps the model deeply explore and aggregate information from unlabeled nodes in the text graph, making fuller use of the labeled information in the text. Financial texts involve domain-specific knowledge, and standardization is difficult and costly. Therefore, training models using self-supervised learning with a limited number of labeled samples is of great significance. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a financial text classification method based on graph contrastive learning. By using graph contrastive learning technology, the model can mine information from unlabeled samples, capture semantic and grammatical relationships in the text, and integrate external knowledge to obtain text representations, thereby improving the text classification effect.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A financial text classification method based on graph contrastive learning, which specifically includes the following steps:

[0008] Step S1: Obtain the text dataset to be processed and perform preprocessing;

[0009] Step S2: Encode the financial documents to be processed;

[0010] Step S3: Determine the relationships between documents based on financial named entities, key phrases, and semantic relationships in the documents, and add documents with the same relationship to the same set;

[0011] Step S4: Construct a text graph based on the documents and their relationships;

[0012] The text graph defines each document as a document node and uses the document encoding obtained in step S2 as the initial feature of the document node; it establishes a correspondence and performs sampling among the document nodes in the set obtained in step S3;

[0013] Step S5: Construct a graph neural network feature propagation model and input the text graph into the graph neural network model for training through comparative learning;

[0014] Building a Graph Neural Network: A graph neural network model is constructed using a two-layer graph convolutional network, with all nodes of the text graph and their feature matrices as the basis. The edge adjacency matrix and edge weight matrix are used as inputs to the model, and ReLU is used as the activation function after the first layer of the graph convolutional network.

[0015] Contrastive learning training: Data augmentation is performed on the graph to obtain two views. Nodes in the graph are used as target samples, corresponding nodes in views different from the target samples are used as positive samples, other nodes in the same view are used as negative samples within the view, and other nodes in different views are used as negative samples between the views. Contrastive loss is calculated and model parameters are optimized.

[0016] Step S6: After inputting the labels into the model for fine-tuning, perform classification prediction on the test set.

[0017] Furthermore, the preprocessing in step S1 includes standardizing the case of English letters, standardizing full-width and half-width characters, removing garbled characters, URLs, and HTML code from the samples.

[0018] Furthermore, step S2 specifically includes: using a BERT pre-trained model to obtain the document [CLS] as the document's encoding, and as the feature matrix of the text graph nodes. .

[0019] Furthermore, step S3 specifically includes: combining annotation models, regular expressions, lexicons, and manual screening to perform named entity recognition and topic key phrase extraction on the text to obtain named entities in the text, such as institution names, financial terms, time and currency phrases, and key phrases; performing semantic role annotation to obtain agent, predicate, and patient; to alleviate dataset imbalance, for categories with fewer than a threshold of samples in the training set, data augmentation is performed to expand to the threshold by randomly replacing the agent and patient; finally, documents with the same phrase are added to the same set.

[0020] Furthermore, the sampling described in step S4 involves randomly selecting N nodes corresponding to each document and connecting them. If the number of documents is less than N, all nodes are connected. Finally, overlapping edges are merged, with the number of edges used as the weight, where N is a natural number.

[0021] Furthermore, the contrastive learning training described in step S5 specifically includes:

[0022] Data augmentation is performed on the original text graph by randomly discarding edges based on weights to transform the graph structure, resulting in two views. These two views are then fed into a graph convolutional network to obtain their encodings. and The document representation is passed through a linear layer and fused with the result of a graph convolutional network to obtain the final document features. ,in These are the linear layer parameters to be learned; using nodes in the view as target samples, corresponding nodes in views different from the target samples as positive samples, other nodes in the same view as negative samples within the view, and other nodes in different views as negative samples between views, the contrastive loss is calculated and the model parameters are optimized.

[0023] The contrastive loss for each target sample is expressed by the following formula:

[0024]

[0025] in, and These represent nodes in the two views respectively. It is the target sample. It is a positive sample. It is a negative sample within the view. These are negative samples between views. It is the temperature coefficient, and sim is the similarity function;

[0026] The mean of the contrast loss over all target samples is used to obtain the contrast loss function:

[0027]

[0028] The model parameters are optimized based on this contrastive loss function.

[0029] Furthermore, the fine-tuning described in step S6 involves using a softmax classifier to calculate the probability of a document belonging to each category, employing the cross-entropy function as the loss function, and using data labels to fine-tune the network.

[0030] The beneficial effects of this invention are as follows: The method of this invention can effectively integrate external knowledge to provide additional semantic guidance for text data and utilize the structural information of text data to make text representations acquire richer and more accurate meanings, helping the model understand proprietary terms in fields such as named entities in financial texts, and completing text classification tasks more efficiently. Furthermore, the method of this invention uses contrastive learning to train the graph neural network model, retaining the key information contained in the graph structure and node representations through edge removal data augmentation while reducing the impact of noise in the data. Self-supervised training helps the model more fully aggregate the information of all nodes and make more full use of unlabeled text data. Attached Figure Description

[0031] Figure 1 This is a flowchart of the method of the present invention;

[0032] Figure 2 This is an overall framework diagram of the present invention. Detailed Implementation

[0033] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification, and the process of achieving the technical effects, thus providing sufficient guidance for implementation. The present invention can be implemented through different specific embodiments, and the details in this specification can also be changed and substituted based on different viewpoints and applications, within the spirit and principles of the present invention.

[0034] See Figure 1 The present invention provides a financial text classification method based on graph contrast learning, which specifically includes the following steps:

[0035] Step S1: Obtain the text dataset to be processed, preprocess it, and divide it into training set, validation set, and test set;

[0036] Step S2: Encode the financial documents to be processed;

[0037] Step S3: Determine the relationships between documents based on financial named entities, key phrases, and semantic relationships in the documents, add documents with the same relationship to the same set, and determine the connection between documents: combine financial domain-related knowledge and semantics to obtain phrases with specific functions in the text, while performing data augmentation on classes with too few samples, and finally add documents with the same phrases to the corresponding set.

[0038] Step S4: Construct a heterogeneous graph of document nodes based on documents and the relationships between them. Documents are treated as nodes in the graph, and each type of relationship between documents is treated as an edge. The document representation obtained in Step S2 is used as the initial feature of the document nodes. Then, sampling and connecting edges are performed on document nodes within the same set. For each pair of nodes with the same relationship for each edge type, if the number is less than N, an edge is connected between each pair of N nodes. If the number is greater than N, the nodes in this group are randomly rearranged and N edges are assigned to each node in turn. After the allocation is completed, the nodes are rearranged and reassigned again until all nodes are assigned N adjacent nodes. Finally, edges of the same type are merged, and the number of edges is used as their weight.

[0039] Step S5: Construct a graph neural network feature propagation model and input the text graph into the graph neural network model. Train the model through contrastive learning. Specifically: Construct the graph neural network: Use a two-layer graph convolutional network to construct a graph neural network model, using all nodes of the text graph and their feature matrices. The edge adjacency matrix and edge weight matrix are used as inputs to the model. ReLU is used as the activation function after the first layer of graph convolutional network. Contrastive learning training: data augmentation is performed on the graph to obtain two views. The nodes in the graph are used as target samples, the corresponding nodes in the views that are different from the target samples are used as positive samples, other nodes in the same view are used as negative samples within the view, and other nodes in different views are used as negative samples between views. The contrastive loss is calculated and the model parameters are optimized.

[0040] Step S6: After inputting the labels into the model for fine-tuning, perform classification prediction on the test set.

[0041] The preprocessing in step S1 specifically includes: unifying the case of English letters, unifying full-width and half-width characters, removing garbled characters, URLs, and HTML code.

[0042] Step S2 specifically includes: obtaining document representations as initial values ​​for document nodes in the text graph, for example, using a BERT pre-trained model to obtain document [CLS] as the document encoding.

[0043] Step S3 specifically includes: combining annotation models, regular expressions, lexicons, and manual screening to perform named entity recognition and topic key phrase extraction on the text to obtain named entities in the text, such as institution names, financial terms, time and currency phrases, and key phrases; performing semantic role annotation to obtain agent, predicate, and patient; to alleviate dataset imbalance, for categories with fewer than a threshold of samples in the training set, data augmentation is performed by randomly replacing the agent and patient to expand to the threshold; finally, documents with the same phrase are added to the same set.

[0044] The specific methods for contrastive learning training in S5 are as follows:

[0045] Data augmentation of the original text graph is performed by randomly discarding edges based on their weights. This method adjusts only relatively uncritical edges in the graph structure without disrupting node representation. Specifically, edge weights are assigned a linearly increasing retention probability. When discarding an edge, a Bernoulli distribution of the corresponding probability is drawn for each edge; 0 indicates discarding the edge, and 1 indicates retention. Two data augmentations are performed to obtain two views. These two views are then fed into a graph neural network to obtain their encodings. and Then, the document representation is passed through a linear layer and fused with the result of the graph convolutional network to obtain the final document features. ,in These are the linear layer parameters to be learned; using nodes in the view as target samples, corresponding nodes in views different from the target samples as positive samples, other nodes in the same view as negative samples within the view, and other nodes in different views as negative samples between views, the InfoNCE contrastive loss is calculated and the model parameters are optimized.

[0046] The contrastive loss for each target sample is expressed by the following formula:

[0047]

[0048] in, and These represent nodes in the two views respectively. It is the target sample. It is a positive sample. It is a negative sample within the view. These are negative samples between views. is the temperature coefficient, and sim is the similarity function.

[0049] The mean of the contrast loss over all target samples is used to obtain the contrast loss function:

[0050]

[0051] The model parameters are optimized based on this contrastive loss function.

[0052] The fine-tuning process in step S6 is as follows: fine-tuning is performed based on the labels in the training set, the probability of a document belonging to each category is calculated using a softmax classifier, the cross-entropy function is used as the loss function, the trained model is obtained after training, and the labels of all data are predicted on the dataset.

[0053] Example

[0054] See Figure 2 This embodiment specifically includes:

[0055] Step S1: Obtain the text dataset. Specifically, the top ten categories of text and event types with the most labels from the CCKS 2022 Few-Shot event extraction evaluation dataset for the financial field are taken as the experimental dataset. The dataset is then divided into training, testing, and validation sets according to a certain ratio. The training and validation sets are of the same size, and the proportion of the testing set is shown below.

[0056] Step S101: Clean the text data. The dataset was crawled from various portals. First, remove irrelevant parts such as HTML tags and URLs.

[0057] Step S2: Obtain text feature encoding. The experiment uses the HuggingFace pre-trained model chinese-roberta-wwm-ext for text encoding, taking the [CLS] token as the document representation to obtain the document feature matrix. ;

[0058] Step S3: Using the corresponding annotation model, vocabulary, and regular expressions, obtain the relationships between data to enhance its graph representation, and enhance the categories with fewer samples:

[0059] Named entities are extracted from the text using a pre-trained Chinese model; key phrases are extracted from the text using a keyword extraction model; and similar words such as time, amount, and stock are merged into a single word using rule merging; semantic role labeling is performed on the text to extract predicates; and the number of samples for fewer sample categories is expanded by replacing named entities and arguments in the training set.

[0060] Step S4: The method for constructing the text graph is as follows: using documents as nodes and employing the document feature matrix. As the initial value of node features; create edges based on the results of step S3. If two documents have the same words, then connect the two documents with an edge. Therefore, there are three types of edges. To control the number of edges in the graph, sampling is required. If for a certain type of edge, the number of connections between a document and other documents exceeds 20, then randomly select 20 edges. Finally, the weight between document nodes is used as the number of heavy edges.

[0061] Step S5: Construct a graph neural network model and train it in a self-supervised manner using contrastive learning. To avoid destroying the text representation vectors, this experiment uses graph structure transformation instead of node representation transformation for data augmentation.

[0062] Step S501: The method for generating two views through data augmentation is to randomly remove some edges. After sorting the edge weights, the larger the edge weight, the lower the probability of it being removed. The probability of removing the edge with the smallest weight is 0.6, and the probability increases in an arithmetic progression as the edge weight increases, with the probability of removing the edge with the largest weight being 0.3. The expected proportion of removed edges to the total number of edges is 0.4. After determining the probability of removing each edge, a Bernoulli distribution is used to decide whether to remove the edge. Two data augmentations are performed on the text graph to obtain two views, which will be described below using... and This represents a node within it.

[0063] Step S502: The graph neural network used is a two-layer graph convolutional network, and the message passing method is as follows:

[0064]

[0065] in Represents a node exist Layer representation; It is the activation function, here it is ReLU; It is a set of relations, and here it refers to the edge type from the steps described above; Represents the set of critical nodes; It is the regularization constant, which is taken here. ; It is a parameter matrix. Let the features of the nodes in the two views after passing through the graph neural network be respectively... and ;

[0066] Step S503: Fuse the output of the graph convolutional network with the linearly transformed document representation to obtain... , as the final feature of the text;

[0067] Step S504: Calculate the contrast loss. The contrast loss for each target sample is:

[0068]

[0069] in, and These represent nodes in the two views respectively. It is the target sample. It is a positive sample. It is a negative sample within the view. These are negative samples between views. It is the temperature coefficient, and sim is the cosine similarity function.

[0070] The mean of the contrast loss over all target samples is used to obtain the contrast loss function:

[0071]

[0072] Based on the contrastive loss function, a learning rate of 0.005 is used, and the network is iterated 20 times. The parameters of the graph convolutional network are optimized using the Adam optimizer.

[0073] Step S6: Based on the relationship between the text and the label category, obtain the predicted classification result of the text. The cross-entropy function is used as the loss function to fine-tune the model. A learning rate of 0.005 is used for two iterations, and the accuracy is finally obtained on the test dataset as the classification evaluation metric. When the data in the test set accounts for 0.9%, 0.7%, and 0.5% of the total data, this invention improves the accuracy by 1.0%, 0.2%, and 0.7%, respectively.

[0074] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Therefore, any simple modifications, equivalent changes, and alterations made to the above implementation methods based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall be included within the scope of the claims of the present invention as equivalent implementations of the present invention.

Claims

1. A financial text classification method based on graph contrastive learning, characterized in that, The method specifically includes the following steps: Step S1: Obtain the text dataset to be processed and perform preprocessing; Step S2: Encode the financial documents to be processed; Step S3: Determine the relationships between documents based on financial named entities, key phrases, and semantic relationships in the documents, and add documents with the same relationship to the same set; Step S4: Construct a text graph based on the documents and their relationships; The text graph defines each document as a document node and uses the document encoding obtained in step S2 as the initial feature of the document node; it establishes a correspondence and performs sampling among the document nodes in the set obtained in step S3; Step S5: Construct a graph neural network feature propagation model and input the text graph into the graph neural network model for training through comparative learning; Building a Graph Neural Network: A graph neural network model is constructed using a two-layer graph convolutional network, with all nodes of the text graph and their feature matrices as the basis. The edge adjacency matrix and edge weight matrix are used as inputs to the model, and ReLU is used as the activation function after the first layer of the graph convolutional network. Contrastive learning training: Data augmentation is performed on the graph to obtain two views. Nodes in the graph are used as target samples, corresponding nodes in views different from the target samples are used as positive samples, other nodes in the same view are used as negative samples within the view, and other nodes in different views are used as negative samples between the views. Contrastive loss is calculated and model parameters are optimized. Step S6: After fine-tuning the model by inputting the labels, perform classification prediction on the test set; where: Step S2 specifically includes: using a BERT pre-trained model to obtain the document [CLS] as the document's encoding, and as the feature matrix of the text graph nodes. ; Step S5, the contrastive learning training, specifically includes: Data augmentation is performed on the original text graph by randomly discarding edges based on weights to transform the graph structure, resulting in two views. These two views are then fed into a graph convolutional network to obtain their encodings. and The document representation is passed through a linear layer and fused with the result of a graph convolutional network to obtain the final document features. ,in These are the linear layer parameters to be learned; using nodes in the view as target samples, corresponding nodes in views different from the target samples as positive samples, other nodes in the same view as negative samples within the view, and other nodes in different views as negative samples between views, the contrastive loss is calculated and the model parameters are optimized. The contrastive loss for each target sample is expressed by the following formula: ; in, and These represent nodes in the two views respectively. It is the target sample. It is a positive sample. It is a negative sample within the view. These are negative samples between views. It is the temperature coefficient, and sim is the similarity function; The mean of the contrast loss over all target samples is used to obtain the contrast loss function: ; The model parameters are optimized based on this contrastive loss function.

2. The financial text classification method based on graph contrast learning according to claim 1, characterized in that, Step S1 preprocessing includes standardizing the case of English letters, standardizing full-width and half-width characters, removing garbled characters, URLs, and HTML code from the samples.

3. The financial text classification method based on graph contrast learning according to claim 1, characterized in that, Step S3 specifically includes: combining annotation models, regular expressions, lexicons, and manual screening to perform named entity recognition and topic key phrase extraction on the text to obtain named entities in the text, such as institution names, financial terms, time and currency phrases, and key phrases; performing semantic role annotation to obtain agent, predicate, and patient; to alleviate dataset imbalance, for categories with fewer than a threshold of samples in the training set, data augmentation is performed by randomly replacing the agent and patient to expand to the threshold; finally, documents with the same phrase are added to the same set.

4. The financial text classification method based on graph contrast learning according to claim 1, characterized in that, The sampling described in step S4 involves randomly selecting N nodes corresponding to each document and connecting them. If the number of documents is less than N, all nodes are connected. Finally, overlapping edges are merged, with the number of edges used as the weight, where N is a natural number.

5. The financial text classification method based on graph contrast learning according to claim 1, characterized in that, The fine-tuning described in step S6 involves using a softmax classifier to calculate the probability of a document belonging to each category, employing the cross-entropy function as the loss function, and using data labels to fine-tune the network.

Citation Information

Patent Citations

  • Heterogeneous information network keyword generation method based on comparative learning pre-training

    CN116662565A

  • Graph-based global semantic perception multi-modal chaffy detection method

    CN117251791A