A group chat financial information demand prediction method

Abstract

In order to obtain demand group financial information with high commercial value in group financial information, the application discloses an algorithm model for demand prediction of group financial information. BiLSTM (bidirectional long short-term memory neural network) and CRF (conditional random field) are used to extract feature words of group financial information, so that the group financial information is preliminarily processed, the feature words are obtained, the duplicate operation is performed to obtain the features of the training GBDT (gradient boosting decision tree) algorithm model, the word frequency digital vector is generated according to the word frequency of the feature words of the group financial information and the features of the GBDT algorithm model, the obtained word frequency digital vector is the training data of the training GBDT algorithm model, the GBDT algorithm model is trained in combination with the labeling of the classification result, and the demand prediction of the group financial information is performed by using the trained GBDT algorithm model. Through experiments, it can be obtained that the accuracy rate of demand prediction of the method reaches 87.3%.

Description

Technical Field

[0001] This invention belongs to the field of group chat financial information prediction, and in particular relates to a method for predicting the demand for group chat financial information based on bidirectional long short-term memory neural networks, conditional random fields, and gradient boosting decision tree algorithms. Background Technology

[0002] The financial industry is a traditional sector, yet also a developing one in my country, closely intertwined with our lives. Its development in recent years has been rapid, creating enormous business opportunities. It's easy to see this in social media platforms like WeChat group chats, where numerous financial messages are exchanged between investors and consumers. These messages are characterized by short texts, with consumers clearly describing their needs in concise language. Another characteristic is that the majority of these group chat messages consist of supply information from investors, while consumer information is minimal. This small amount of demand information represents a significant business opportunity; uncovering this demand information can provide suppliers with customers, possessing substantial commercial value. Currently, recommendation and prediction systems are applied in many fields, such as Taobao and Toutiao. These systems can predict products users are likely to purchase and push them to them, as well as articles and videos users might like. These recommendation and prediction systems can bring significant commercial value to businesses, and with the increasing maturity of the internet, they have a promising future. However, current predictive algorithms and methods used in financial industry research rarely address the prediction of demand-driven financial information. Most studies focus on predicting price trends of financial products, financial market trends, and financial time series data. If predictive algorithms could be applied to predicting demand-driven financial information, more scarce demand-driven financial information could be uncovered, allowing for the creation of higher commercial value from this information.

[0003] The existing method for preprocessing financial information data in group chats is to first clean the data and then vectorize the Chinese text. Data cleaning first removes symbols, including punctuation, from the text data to obtain pure Chinese text. This is generally done using the `compile` function in the `re` module of the Python programming language, which utilizes regular expressions. Then, word segmentation is used to segment the pure Chinese financial information in group chats. This segmentation mostly relies on the word segmentation tool "jieba" in Python. After segmentation, stop words are removed using a stop word list. The result of data cleaning is the words in the financial information from group chats. However, these words cannot be directly recognized by the algorithm model and still need to be vectorized. Chinese text vectorization converts the existing text data into numerical vectors. These numerical vectors are the data that the algorithm model can use for training. Each piece of financial information from group chats corresponds to an n-dimensional numerical vector. The dimension of this vector depends on the number of features in the financial information from group chats. These features are the words obtained after data cleaning. The n-dimensional vector corresponds to n features, representing the values ​​of a particular piece of financial information from group chats under the n features. One of the existing tools for Chinese text vectorization is one-hot encoding. One-hot encoding uses the words in the segmented text data after removing repetitions as features. When one-hot encoding a text data, the existing features are marked as 1, and the remaining features are marked as 0. However, directly using one-hot encoding to process group chat financial information to form numerical vectors faces problems such as low efficiency in generating numerical vectors and poor quality of the resulting numerical vectors in demand prediction of group chat financial information, which seriously reduces the accuracy of the algorithm model. When the data volume is large, such as 20,000 group chat financial information messages, one-hot encoding will generate many features, which will lead to a very large dimensional vector, even thousands of dimensions. However, there are very few features that can be marked as 1 for a single data point, which increases the difficulty of training the model and affects the accuracy of the algorithm model. Secondly, some words that have little contribution to classification are also used as features for training the algorithm model. For example, words such as "here" and "exist" have little contribution to the model's classification training. Adding these words as features to the model training will increase the model training time and reduce the accuracy of the algorithm model in predicting demand for group chat financial information. In deep learning neural network algorithm models, word embedding is also a way to transform group chat financial information into model training data. However, due to the scarcity of demand group chat financial information, the training data obtained by using word embedding is very ineffective in predicting demand group chat financial information. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies in obtaining effective features from preprocessed group chat financial information and the low accuracy of algorithm models in predicting demand group chat financial information. It provides a method for predicting demand group chat financial information based on bidirectional long short-term memory neural networks, conditional random fields, and gradient boosting decision tree algorithms.

[0005] The present invention adopts the following technical solution:

[0006] The BiLSTM (Bi-directional Long Short-Term Memory) and CRF (Conditional Random Field Algorithm) models, i.e., the BiLSTM+CRF model, are used to process financial information in group chats to obtain feature words. Based on the word frequency of these feature words, vectorization is performed. Subsequently, the GBDT (Gradient Boosting Decision Tree) algorithm is used for training and prediction, including the following steps:

[0007] Step 1: Classify and label the group chat financial information. Label demand-type group chat financial information as 1 and supply-type group chat financial information as 0. Select a portion of the group chat financial information for training the bidirectional long short-term memory neural network and conditional random field model. The number of group chat financial information selected depends on the number of group chat financial information, and the number selected is between 10% and 20%. For example, if there are 1,000 group chat financial information, 200 can be selected, and if there are 10,000 group chat financial information, 1,000 can be selected. Use business type words (e.g., loan, financing), institution type words (e.g., bank, company), contact type words (e.g., contact, telephone), and verbs expressing emotion related to business (e.g., need, provide) as feature words. When labeling the feature words of the group chat financial information, use the "BIO" (Begin Inside Outside) labeling system. Label the group chat financial information used to train the bidirectional long short-term memory neural network and conditional random field model with feature words, and use the labeled data as the training dataset for training the bidirectional long short-term memory neural network and conditional random field model.

[0008] Step two: Using the training dataset obtained in Step one for training the bidirectional long short-term memory neural network and conditional random field model, the training dataset is divided into a first training set and a first test set. In this invention, the division is carried out in an 8:2 ratio. The first training set is used to train the bidirectional long short-term memory neural network and conditional random field model, and the first test set is used to test the trained bidirectional long short-term memory neural network and conditional random field model. Then, the bidirectional long short-term memory neural network and conditional random field model are trained and tested using the first training set and the first test set. The trained bidirectional long short-term memory neural network and conditional random field model are used to predict the feature words of the group chat financial information to obtain the first feature words. The result of the prediction processing of each piece of group chat financial information is: extracting the feature words contained in the original complex group chat financial information.

[0009] Step 3: After removing duplicate words from the first feature words obtained in Step 2, use them as the bag of features for training the gradient boosting decision tree algorithm model. Assuming there are n features, encode the group chat financial information based on the word frequency of the feature words corresponding to each piece of group chat financial information in the bag of features. After encoding, each piece of group chat financial information is converted into a corresponding n-dimensional word frequency vector. The dimension of the word frequency vector is the same as the number of features in the gradient boosting decision tree algorithm model.

[0010] Step four: Before training the gradient boosting decision tree model, the n-dimensional word frequency vectors obtained in step three need to be classified into two categories: a second training set and a second test set. The second training set is used to train the gradient boosting decision tree algorithm model, and the second test set is used to test the trained gradient boosting decision tree algorithm model. In this invention, the ratio of the second training set to the second test set is set to 8:2. The gradient boosting decision tree algorithm model is trained using the second training set and the corresponding annotations of the classification results described in step one, resulting in a trained gradient boosting decision tree algorithm model. The trained gradient boosting decision tree algorithm model is then used to perform classification prediction on the second test set. The accuracy of the gradient boosting decision tree algorithm model is obtained by comparing and analyzing the prediction results with the corresponding annotations of the classification results described in step one.

[0011] The beneficial effects of this invention are as follows: Compared with existing technologies, this invention uses a bidirectional long short-term memory neural network and a conditional random field to filter words in group chat financial information to obtain feature words with high training value for the algorithm model, thereby improving the quality of the group chat financial information dataset and effectively shortening the training time of the prediction algorithm model; combined with the gradient boosting decision tree algorithm model for demand prediction of group chat financial information, experiments show that the accuracy of demand prediction by this method reaches 87.3%, compared with the current existing demand prediction accuracy of 68.4% based on CNN (Convolutional Neural Networks), indicating that this is a reliable method for demand prediction of group chat financial information. Attached Figure Description

[0012] Appendix Figure 1 This is a flowchart for tagging characteristic words in financial information from group chats.

[0013] Appendix Figure 2 This is a flowchart illustrating the training, testing, and use of the trained BiLSTM+CRF model to predict feature words in group chat financial information.

[0014] Appendix Figure 3 This is a flowchart for converting financial information from group chats into word frequency vectors.

[0015] Appendix Figure 4 This is a flowchart of training the GBDT algorithm and testing its accuracy using the word frequency numerical vectors of financial information from group chats.

[0016] Appendix Figure 5 This is the overall flowchart, which describes the prediction and processing of feature words in group chat financial information, the training of the GBDT algorithm model, and its prediction. Detailed Implementation

[0017] Step 1: Classify and label the group chat financial information. Label demand-type group chat financial information as 1 and supply-type group chat financial information as 0. Select a portion of the group chat financial information for training the bidirectional long short-term memory neural network and conditional random field model. The number of group chat financial information selected depends on the number of group chat financial information, and the number selected is between 10% and 20%. For example, if there are 1,000 group chat financial information, 200 can be selected, and if there are 10,000 group chat financial information, 1,000 can be selected. Use business type words (e.g., loan, financing), institution type words (e.g., bank, company), contact type words (e.g., contact, telephone), and verbs expressing emotion related to business (e.g., need, provide) as feature words. When labeling the feature words of the group chat financial information, use the "BIO" (Begin Inside Outside) labeling system. Label the group chat financial information used to train the bidirectional long short-term memory neural network and conditional random field model with feature words, and use the labeled data as the training dataset for training the bidirectional long short-term memory neural network and conditional random field model.

[0018] The specific implementation process of the embodiment is as follows:

[0019] Select a portion of financial information from group chats and annotate the characteristic words based on the characteristic words in Step 1. The annotation adopts the "BIO" system, where B-ins represents the beginning of the characteristic word, I-ins represents the middle to the end of the characteristic word, and O represents non-characteristic words.

[0020] Suppose there are 5000 financial messages in a group chat, 500 of them can be selected for training a bidirectional long short-term memory neural network and a conditional random field.

[0021] One group chat message reads: "Zhengda Financial Holdings Co., Ltd.: A powerful financial provider, sincerely seeking high-quality real economy projects nationwide. We welcome capable business partners from all over the country to discuss cooperation."

[0022] This group chat financial information is classified as supply-type, therefore the classification result for this group chat financial information is 0. The characteristic words for this group chat financial information are labeled as follows:

[0023] Table 1: Annotation of Characteristic Keywords in Financial Information from Group Chats

[0024] just big gold melt have limit male manage ： Reality force Capital gold O O O O O O B-ins I-ins O O O B-ins I-ins square ， Sincerity Seeking Complete country excellent quality Reality body item Head ， O O B-ins I-ins O O O O O O B-ins I-ins O joyous welcome Complete country have able force of business Service right catch people B-ins I-ins O O O O O O O O O O O scholar Negotiation talk combine do 。 O B-ins I-ins B-ins I-ins O

[0025] Select an appropriate amount of group chat financial information for training the BiLSTM+CRF model, and preprocess the group chat financial information by labeling feature words to obtain the training dataset. This will be a one-time process; as the amount of training dataset increases, the accuracy of the BiLSTM+CRF model in predicting feature words will become higher and higher.

[0026] Step two: Using the training dataset obtained in Step one for training the bidirectional long short-term memory neural network and conditional random field model, the training dataset is divided into a first training set and a first test set. In this invention, the division is carried out in an 8:2 ratio. The first training set is used to train the bidirectional long short-term memory neural network and conditional random field model, and the first test set is used to test the trained bidirectional long short-term memory neural network and conditional random field model. Then, the bidirectional long short-term memory neural network and conditional random field model are trained and tested using the first training set and the first test set. The trained bidirectional long short-term memory neural network and conditional random field model are used to predict the feature words of the group chat financial information to obtain the first feature words. The result of the prediction processing of each piece of group chat financial information is: extracting the feature words contained in the original complex group chat financial information.

[0027] The specific implementation process of the embodiment is as follows:

[0028] Using Python 3.6, the BiLSTM model was built using the Torch framework. The word vector dimension `max_len` was set to 50, and dropout was set to 0.5 to prevent overfitting. The CRF model adopted a tri-gram format, and the `sklearn-crfsuite` library was used to import the model, employing the `sklearn-crfsuite.CRF` method. To ensure the accuracy of the obtained CRF model, the parameters were set as follows: the lbfgs algorithm was chosen for its space and computational efficiency; `c1` and `c2` were set to 0.1; the maximum number of iterations `max_iterations` was set to 100; and `all_possible_transitions` was set to True. The BiLSTM model was trained on the first training set using the `train` method, and its output was used as the input to the CRF model for further training. After the BiLSTM+CRF model is trained, its performance is evaluated using the first test set. Finally, the BiLSTM+CRF model is used to predict the feature words of the financial information in the group chat to obtain the first feature words.

[0029] Imagine a request in a group chat about finance: the person wants to collaborate with friends who have funds for mutual benefit.

[0030] The trained BiLSTM+CRF model is used to predict the feature words of this group chat financial information. The final prediction result is the feature words contained in this group chat financial information, and the output prediction result is: 'fund cooperation'.

[0031] Step 3: After removing duplicate words from the first feature words obtained in Step 2, use them as the bag of features for training the gradient boosting decision tree algorithm model. Assuming there are n features, encode the group chat financial information based on the word frequency of the feature words corresponding to each piece of group chat financial information in the bag of features. After encoding, each piece of group chat financial information is converted into a corresponding n-dimensional word frequency vector. The dimension of the word frequency vector is the same as the number of features in the gradient boosting decision tree algorithm model.

[0032] The specific implementation process of the embodiment is as follows:

[0033] Step two processes yielded the feature words for all group chat financial messages. These feature words were stored in a list called `words`, and duplicate words were removed using Python. A bag-of-words (vocab) was set up to receive the removed words. The specific statement was: `vocab = sorted(set(words), key = words.index)`, where `sorted` is the sorting function, the object to be sorted is the list of words containing feature words, and `key` is a function with one parameter used to extract the comparison value for each element. The value of `key` is `words.index`, which is the index of the list of words. A two-dimensional array `count` was initialized. A for loop was used to iterate through each group chat financial message processed in step two. Each feature word A in the group chat financial message was compared with the words in the bag-of-words `vocab`. The position of feature word A in the bag-of-words was found and recorded as `count[i][pos] = count[i][pos] + 1`, where `i` is the i-th group chat financial message, and `pos` is the position of feature word A in the bag-of-words. Setting `count[i][pos] = count[i][pos] + 1` is equivalent to counting the word frequency of the feature word corresponding to the group chat financial message. The `DataFrame` function in the pandas library of Python is used to generate a term frequency vector `countVector`. The specific implementation of `countVector` is: `countVector = DataFrame(count, columns = vocab)`. The assigned `count` vector is passed into the `DataFrame` function, and the columns of the `countVector` term frequency vector are set to the bag-of-words `vocab`, with each piece of group chat financial information corresponding to one row. This yields the term frequency vectors corresponding to the group chat financial information generated in step two. The dimension of the term frequency vector is the number of feature words in the bag-of-words `vocab`.

[0034] Assuming there are three financial messages in a group chat, the characteristic words obtained after step two are shown in the table below:

[0035] Table 2: List of Featured Terms

[0036]

[0037] The three group chat messages containing financial information, after deduplication, have the following nine characteristic words: funds, projects, seeking, cooperation, investment, entity, need, support, and hotel. These nine characteristic words are added to a bag-of-words (vocab) as the dimension of the word frequency vector. These characteristic words will also be used as features for training the GBDT algorithm model.

[0038] The generated word frequency vectors are shown in the table below:

[0039] Table 3: Frequency Vectors of Financial Information in Group Chats

[0040]

[0041] The number in the word frequency vector generated for each financial message in a group chat represents the number of times that financial message contains this feature word. Specifically, the word frequency vector for financial message three in group chat is [1,0,0,0,0,1,1,1,1].

[0042] Step four: Before training the gradient boosting decision tree model, the n-dimensional word frequency vectors obtained in step three need to be classified into two categories: a second training set and a second test set. The second training set is used to train the gradient boosting decision tree algorithm model, and the second test set is used to test the trained gradient boosting decision tree algorithm model. In this invention, the ratio of the second training set to the second test set is set to 8:2. The gradient boosting decision tree algorithm model is trained using the second training set and the corresponding annotations of the classification results described in step one, resulting in a trained gradient boosting decision tree algorithm model. The trained gradient boosting decision tree algorithm model is then used to perform classification prediction on the second test set. The accuracy of the gradient boosting decision tree algorithm model is obtained by comparing and analyzing the prediction results with the corresponding annotations of the classification results described in step one.

[0043] The specific implementation process of the embodiment is as follows:

[0044] When obtaining the GBDT algorithm model, the `GradientBoostingClassifier` method from the `sklearn.ensemble` library provided by the Python programming language is used. The specific values ​​for the relevant parameters in the `GradientBoostingClassifier` method are: initial sub-model `init` is None, learning rate (reduction) is 0.1, loss function is `deviance`, maximum depth `max_depth` is 3, maximum number of features involved in node splitting `max_features` is None, maximum number of leaf nodes `max_leaf_nodes` is None, minimum number of leaf node samples `min_simples_leaf` is 1, minimum number of samples required for splitting `min_simples_split` is 2, minimum total weight of leaf node samples `min_weight_fraction_leaf` is 0.0, number of sub-models `n_estimators` is 100, random state of the random machine object is None, subsample rate is 1.0, log verbose is 0, and whether to perform a warm start (if so, the next training will be performed as an append-to-tree) `warm_start` is False. The established GBDT algorithm model is trained using the `fit` method, which requires two objects: a second training set and the labeled classification results of the group chat financial information corresponding to the second training set. The accuracy of the trained GBDT algorithm model is evaluated using the `predict` method, which takes the second test set object as input. Using the `metrics` method from the Sklearn library, the `accuracy_score` function is used to evaluate the accuracy of the GBDT algorithm model. The `accuracy_score` function has two objects: the predicted classification results of the second test set output by the `predict` method and the labeled classification results of the group chat financial information corresponding to the second test set. The `recall_score` function is used to evaluate the recall of the GBDT algorithm model. The `recall_score` function has three objects: two are the same as those in the `accuracy_score` function, and the third object is the average value of the string data, with an `average` type of `macro`.

[0045] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for forecasting the demand for financial information in group chats, characterized in that, Includes the following steps: Step 1: Classify and label the financial information in the group chat, labeling demand-type group chat financial information as 2 and supply-type group chat financial information as 3; select a portion of the group chat financial information for training the Conditional Random Field (CRF) model. Based on the textual characteristics of financial information in group chats, the "BIO" system is used to classify the words in the financial information in group chats used to train the conditional random field model into entity types. The classified financial information in group chats is used as the training data for training the conditional random field model, and is denoted as the training dataset. Step 2: Divide the training dataset obtained in Step 1 into a first training set and a first test set in an 8:2 ratio. The first training set is used to train the Conditional Random Field (CRF) model, and the first test set is used to test the trained CRF model. After completing the training and testing of the CRF model using the first training set and the first test set, use the trained CRF model to filter feature words from all collected group chat financial information to obtain the first feature words with high predictive value. Step 3: Remove duplicates from the first feature words in Step 2 to obtain a set of feature words without redundancy; based on this set of feature words, convert each piece of financial information in the group chat into its corresponding digital vector using one-hot encoding, wherein the dimension of the digital vector is consistent with the number of words in the feature word set; Step four: Divide the numerical vectors obtained in step three into a second training set and a second test set in an 8:2 ratio; train the Gradient Boosting Decision Tree (GBDT) model using the second training set and the corresponding group chat financial information classification results annotations; use the trained Gradient Boosting Decision Tree model to perform classification prediction on the second test set, and verify the model's prediction accuracy by comparing the prediction results with the corresponding classification results annotations on the second test set; finally, use the trained Gradient Boosting Decision Tree model to perform demand classification prediction of group chat financial information.

2. The demand forecasting method for group chat financial information according to claim 1, characterized in that, The characteristic words in step two include business type words, institution type words, connection type words, and verbs related to business and expressing emotion in group chat financial information; among them, business type words refer to words that reflect the core content of financial business, institution type words refer to words that represent the main participants in financial business, connection type words refer to words that reflect the interactive intention of the information publisher, and verbs related to business and expressing emotion refer to verbs that reflect the information demand or supply tendency.

3. The demand forecasting method for group chat financial information according to claim 1, characterized in that, In step one, the entity types of the "BIO" system are divided into five categories: B-ins, I-ins, B-inf, I-inf, and O. Among them, ins represents the organization name, inf represents the words that provide business information and needs, B represents the beginning of the named entity, I represents the middle to the end of the named entity, and O represents other types.

4. The method for predicting the demand for financial information in group chats according to claim 1, characterized in that, In step two, the training parameters of the conditional random field model are set as follows: the algorithm uses the lbfgs algorithm, c1 is 0.1, c2 is 0.1, the maximum number of iterations is 100, and all possible transformation parameters are set to True.

5. The method for predicting the demand for financial information in group chats according to claim 1, characterized in that, In step four, the training parameters for the gradient boosting decision tree model are set as follows: initial sub-model is None, learning rate is 0.1, loss function is deviance, maximum depth is 3, maximum number of features involved in the decision when splitting a node is None, maximum number of leaf nodes is None, minimum number of samples for a leaf node is 1, minimum number of samples required for splitting is 2, total weight of minimum samples for a leaf node is 0.0, number of sub-models is 100, randomization is None, subsampling rate is 1.0, log redundancy is 0, and warm start parameter is False.

Images