Text classification methods, devices, computer equipment, and storage media in the financial field

Abstract

This application relates to the fields of natural language processing and fintech, providing a method, apparatus, computer device, and storage medium for text classification in the financial field. The method includes: acquiring text to be classified; inputting the text into a pre-built hybrid neural network model, causing the hybrid neural network model to perform the following processing steps: converting the text into sentence vectors through a model encoding layer; performing convolution processing on the sentence vectors through a feature attention layer to obtain character key vectors and character vectors based on the text, and weighting the character key vectors and character vectors to obtain a feature weighted vector; performing convolution processing on the feature weighted vectors through a text classification layer to obtain feature vectors, and outputting the text's category probability based on the feature vectors; obtaining the text's category probability; and determining whether the text belongs to the financial field based on the text's category probability. This method can improve the recognition accuracy of text in the financial field.

Description

Technical Field

[0001] This application relates to the fields of natural language processing technology and financial technology technology, and in particular to a text classification method, apparatus, computer equipment, storage medium and computer program product in the financial field. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, the mainstream method for text classification in the financial field has shifted from domain dictionary-based methods to deep learning-based methods.

[0003] However, existing deep learning-based text classification methods in the financial field can only extract shallow features of the text and cannot consider the overall semantic information of the sentence and the corresponding local feature information. Moreover, most methods rely on pre-trained language models and cannot accurately identify financial text. Summary of the Invention

[0004] Therefore, it is necessary to provide a text classification method, apparatus, computer equipment, computer-readable storage medium, and computer program product in the financial field to address the aforementioned technical problems.

[0005] Firstly, this application provides a text classification method in the financial field, including:

[0006] Get the text to be categorized;

[0007] The text is input into a pre-built hybrid neural network model, which then performs the following processing steps: The text is converted into a sentence vector through a model encoding layer; a feature attention layer is used to perform a convolution operation on the sentence vector to obtain a character key vector, and a character vector is obtained based on the text; the character key vector and the character vector are weighted together to obtain a feature weight vector; a text classification layer is used to perform a convolution operation on the feature weight vector to obtain a feature vector, and the category probability of the text is output based on the feature vector.

[0008] Obtain the category probability of the text;

[0009] Based on the category probability of the text, determine whether the text belongs to the financial field.

[0010] In one embodiment, inputting the text into a pre-built hybrid neural network model includes:

[0011] The meaning of words in the text is encoded through a word embedding layer;

[0012] The position of the word is represented by a positional encoding layer to obtain the text in vector form;

[0013] The text in vector form is input into a pre-built hybrid neural network model.

[0014] In one embodiment, the model encoding layer includes a multi-head attention mechanism and a forward propagation layer. The process of converting the text into sentence vectors through the model encoding layer includes:

[0015] The text is input into the multi-head attention mechanism to obtain a concatenated vector;

[0016] The concatenated vector is input into the forward propagation layer for nonlinear transformation to obtain the sentence vector.

[0017] In one embodiment, the convolution operation on the sentence vector to obtain the character key vector includes:

[0018] Perform a convolution operation on the sentence vector to obtain a grammatical feature information vector;

[0019] The character key vector is obtained by performing max pooling on the grammatical feature information vector.

[0020] In one embodiment, obtaining the character vector based on the text includes:

[0021] Obtain the text;

[0022] The text is input into a bidirectional gating unit for encoding to obtain a character vector.

[0023] In one embodiment, outputting the category probability of the text based on the feature vector includes:

[0024] Max pooling is performed on the feature vector to obtain the deep features of the text;

[0025] Based on the deep features of the text, output the category probability of the text.

[0026] In one embodiment, outputting the category probability of the text based on its deep features includes:

[0027] The deep features of the text are input into a fully connected layer to obtain the output result;

[0028] Based on the output and the normalized exponential function, the category probability of the text is output.

[0029] Secondly, this application also provides a text classification device in the financial field, comprising:

[0030] The text acquisition module is used to acquire the text to be classified.

[0031] The category probability prediction module is used to input the text into a pre-built hybrid neural network model, so that the hybrid neural network model performs the following processing steps: The text is converted into a sentence vector through a model encoding layer; a character key vector is obtained by convolving the sentence vector through a feature attention layer, and a character vector is obtained based on the text; the character key vector and the character vector are weighted together to obtain a feature weighted vector; a feature vector is obtained by convolving the feature weighted vector through a text classification layer, and the category probability of the text is output based on the feature vector.

[0032] The category probability acquisition module is used to acquire the category probability of the text.

[0033] The financial text judgment module is used to determine whether the text belongs to the financial field based on the category probability of the text.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the above-described method.

[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, the computer program being executed by a processor using the methods described above.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that is executed by a processor using the methods described above.

[0037] The aforementioned text classification method, apparatus, computer equipment, storage medium, and computer program product in the financial field acquire the text to be classified; input the text into a pre-built hybrid neural network model, so that the hybrid neural network model performs the following processing steps: through the model encoding layer, the text is converted into sentence vectors; through the feature attention layer, the sentence vectors are convolved to obtain character key vectors and character vectors are obtained based on the text; the character key vectors and character vectors are weighted to obtain a feature weighted vector; through the text classification layer, the feature weighted vectors are convolved to obtain feature vectors, and the category probability of the text is output based on the feature vectors; the category probability of the text is obtained; and based on the category probability of the text, it is determined whether the text belongs to the financial field. This case involves weighting the character keyness vector and the character vector to obtain a feature weight vector, which enhances the weight expression of key characters in the sentence, thereby considering the local feature information of the sentence. Using a text classification layer, and with the help of convolution operations, the feature weight vector obtained after feature weighting is mined to obtain a feature vector that can represent deep text features, so as to focus on the overall semantic information of the sentence. Based on the feature vector, the category probability of the text is output, thereby determining whether the text belongs to the financial field and improving the recognition accuracy of financial text. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating the application environment of a text classification method in the financial field in one embodiment.

[0040] Figure 2 This is a flowchart illustrating a text classification method in the financial field in one embodiment;

[0041] Figure 3 Here is a diagram of the hybrid neural network model structure in one embodiment;

[0042] Figure 4 Here is a structural diagram of a multi-head attention mechanism in one embodiment;

[0043] Figure 5 This is a schematic diagram illustrating the extraction of convolutional features in one embodiment;

[0044] Figure 6 This is a structural block diagram of a text classification device in the financial field in one embodiment;

[0045] Figure 7This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0048] This application provides a text classification method in the financial field. This embodiment can be executed by a computer device, such as... Figure 1 As shown, a computer device can acquire text to be classified, input the text into a pre-built hybrid neural network model to obtain the text's category probability, and determine whether the text belongs to the financial field based on the category probability. It is understood that this computer device can be implemented through a server, a terminal, or an interactive system between a terminal and a server. In this embodiment, the method includes... Figure 2 The steps shown are as follows:

[0049] Step S201: Obtain the text to be classified.

[0050] Text to be classified refers to text whose domain has not been determined.

[0051] Step S202: Input the text into a pre-built hybrid neural network model.

[0052] After the text is input into the hybrid neural network model, the model performs the following processing steps: The text is converted into sentence vectors through the model encoding layer; the sentence vectors are convolved to obtain character key vectors through the feature attention layer, and character vectors are obtained based on the text; the character key vectors and character vectors are weighted to obtain a feature weighted vector; the feature weighted vectors are convolved to obtain a feature vector through the text classification layer, and the text category probability is output based on the feature vector.

[0053] The specific structure of the hybrid neural network model is as follows: Figure 3As shown, it includes a model encoding layer, a feature attention layer, and a text classification layer. The model encoding layer may include a multi-head attention mechanism and a feedforward layer; the feature attention layer may include a bidirectional gated unit (BiGRU), convolutional operations (Conv), and a max pooling layer; and the text classification layer may include convolutional operations, a max pooling layer, a fully connected layer (Linear), and a normalized exponential function (Softmax).

[0054] After the computer device inputs text into a pre-built hybrid neural network model, the meaning of words in the text is encoded through a word embedding layer, and the position of words is represented through a positional encoding layer, resulting in text in vector form. The text vector is then input into the model's encoding layer, where a multi-head attention mechanism is used to obtain a concatenated vector. This concatenated vector undergoes a combination of residual connection and layer normalization operations (Add&Norm), which is then input into a forward propagation layer for nonlinear transformation to obtain a sentence vector. This sentence vector is then input into a feature attention layer, where a convolution operation is performed to obtain a character key vector, and a character vector is derived from the text. Assuming the character key vector can be expressed as Y', the sentence vector encoded by the bidirectional gating unit can be expressed as B, where B∈R. (c×e) Where c represents the maximum number of characters in the sentence and e is the dimension of the encoded word vector, the specific steps to obtain the feature weighted vector by weighting the character key vector and the character vector are as follows: perform a dot product operation between the sentence vector B and Y', and the calculation process is as shown in formula (1).

[0055] BA=Y'⊙B (1)

[0056] In formula (1), ⊙ represents the weighting operation of key features in the sentence using weighted encoding, and BA represents the character vector after weighting the key features.

[0057] The weighted feature vector is input into the text classification layer, and a convolution operation is performed on the weighted feature vector to obtain the feature vector. Based on the feature vector, the class probability of the text is output.

[0058] Step S203: Obtain the category probability of the text.

[0059] The probability of a text category can include the probability that the text belongs to each category, where a certain category corresponds to the financial field.

[0060] Step S204: Determine whether the text belongs to the financial field based on the category probability of the text.

[0061] After obtaining the category probabilities of the text, the category probability corresponding to the financial field can be determined. When the category probability is greater than a threshold, the text is determined to belong to the financial field; when the category probability is less than the threshold, the text is determined not to belong to the financial field. The threshold can be set according to the actual situation.

[0062] In the aforementioned text recognition method for the financial field, a weighted feature vector is obtained by weighting the character keyness vector and the character vector, which enhances the weight expression of key characters in the sentence and thus considers the local feature information of the sentence. Using a text classification layer, the weighted feature vector obtained by feature weighting is mined through convolution operation to obtain a feature vector that can represent deep text features, so as to focus on the overall semantic information of the sentence. Based on the feature vector, the category probability of the text is output to determine whether the text belongs to the financial field, thereby improving the recognition accuracy of financial text.

[0063] In one embodiment, inputting text into a pre-built hybrid neural network model includes: encoding the meaning of words in the text through a word embedding layer; representing the position of words through a position encoding layer to obtain text in vector form; and inputting the text in vector form into the pre-built hybrid neural network model.

[0064] Word embedding is a general term for language modeling and representation learning techniques in natural language processing. Conceptually, word embedding refers to embedding a high-dimensional space with the number of all words into a continuous vector space with a much lower dimension, where each word or phrase is mapped to a vector in the real number field.

[0065] Location encoding describes the position of an entity in a sequence so that a unique representation is assigned to each position.

[0066] In this embodiment, the meaning of words in the text is encoded through a word embedding layer, and the position of words is represented through a positional encoding layer, resulting in text in vector form. This vector text is then input into a pre-built hybrid neural network model. Converting the input text into vector form facilitates processing of the input text by the hybrid neural network model.

[0067] In one embodiment, the model encoding layer includes a multi-head attention mechanism and a forward propagation layer. The model encoding layer converts text into sentence vectors. The specific steps are as follows: input the text into the multi-head attention mechanism to obtain a concatenated vector; input the concatenated vector into the forward propagation layer for nonlinear transformation to obtain a sentence vector.

[0068] Multi-head attention is an extension of the attention mechanism that can extract information more effectively when processing sequential data. In standard attention mechanisms, a weighted context vector is computed to represent the information of the input sequence. In multi-head attention, multiple sets of attention weights are used, each set of weights can learn different semantic information, and each set of weights generates a context vector. Finally, these context vectors are concatenated and subjected to a linear transformation to obtain the final output.

[0069] The forward propagation layer refers to the process of obtaining the output result by weighted summation of the input data and activation function calculations of the neurons in each layer.

[0070] The specific structure of the multi-head attention mechanism is as follows: Figure 4 As shown, the computer device inputs text into the multi-head attention mechanism to obtain three key vectors: Q vector, K vector, and V vector, where Q vector represents the query vector, K vector represents the key vector, and V vector represents the value vector. The calculation methods for the three key vectors are shown in formulas (2) to (4). At the same time, the multi-head attention mechanism calculates word vectors in the text as shown in formulas (2) to (6).

[0071] q i =w q a i (2)

[0072] k i =w k a i (3)

[0073] v i =w v a i (4)

[0074]

[0075] b i =∑ j softmax(a i,j )v j (6)

[0076] Where, q i Let k represent the Q-vector input to the i-th attention head. i Let v represent the K vector input to the i-th attention head. i Let V be the input vector to the i-th attention head, and w be the vector of choice. q w k w v Let a represent the weight matrix. i,j Let d represent the attention weight of the i-th attention head at the j-th position in the input sequence. kb represents the scaling factor. i Let represent the bias vector of the i-th attention head.

[0077] During the calculation, the K vector and Q vector can be represented as: K = [k1, k2, k3, ..., k n ] and Q = [q1,q2,q3,…,q n ], where n represents the number of characters in the sentence. Therefore, the calculation process of the self-attention mechanism is shown in equation (7).

[0078]

[0079] After obtaining the output vector of the self-attention mechanism, the multi-head attention mechanism is constructed. During the construction process, the output vectors of multiple self-attention mechanisms are combined and concatenated, and a fully connected layer is used to transform the concatenated vector into an output of a specific dimension. Figure 4 The calculation process of the multi-head attention mechanism can be represented by formulas (8) and (9), where formula (8) represents the calculation method of a single self-attention mechanism.

[0080] head i =attention(Kw i K Qw i Q Vw i V (8)

[0081] After obtaining the output vector of a single self-attention mechanism, the output vectors of the single self-attention mechanism are vertically connected to obtain the final joint output result, which is calculated as shown in formula (9).

[0082] MHead(K,Q,V)=Cat(head1,head2,…,head n W Τ (9)

[0083] In equations (8) and (9), w i K w i Q w i V and W Τ Let represent the weight matrix, n represent the number of self-attention mechanisms, and Cat represent the vector concatenation operation.

[0084] Next, normalization is applied to the resulting concatenated vector. Normalization makes the hidden layer data representation of the entire model algorithm more stable.

[0085] After the data is normalized, it is input into the forward propagation layer for nonlinear transformation to obtain the sentence vector. The calculation process is shown in Equation (10).

[0086] FFN(X) = max(0, XW) m1 +b m1 W m2 +b m2 (10)

[0087] In equation (10), the linear rectified function (RELU) is chosen as the activation function, W m1 and W m2 Let b represent the weight matrix. m1 and b m2 This represents the bias parameter.

[0088] In this embodiment, by inputting the text into a multi-head attention mechanism to obtain a concatenated vector, the information of the text can be effectively extracted; by inputting the concatenated vector into the forward propagation layer for nonlinear transformation, a sentence vector is obtained, realizing the layer-by-layer processing and information transmission of the input text.

[0089] In one embodiment, a convolution operation is performed on the sentence vector to obtain a character key vector, specifically including the following steps: performing a convolution operation on the sentence vector to obtain a grammatical feature information vector; performing a max pooling operation on the grammatical feature information vector to obtain a character key vector.

[0090] Convolution is one of the core operations of convolutional neural networks. It extracts features from the input data by performing convolution operations on the input data and the convolution kernel.

[0091] Max pooling is a common operation in convolutional neural networks used to reduce the spatial size of feature maps while preserving the most salient features.

[0092] like Figure 5 As shown, the text is encoded through a word embedding layer, and then a sentence vector is obtained through a model encoding layer. Assume that the sentence vector obtained through the model encoding layer can be represented as: S, S∈R (c×e) For a sentence S, it can be obtained by convolution operation using a convolution kernel of width k: S = {S (1:k) S (2:k+1) S (c-k+1:c)}, where S (c-k+1:c) Let S represent the subsequence from the c-k+1th element to the cth element. After a single convolution operation, sentence S yields the k-gram syntactic feature information Y, as shown in formula (11).

[0093] Y = [y1, y2, y3, ..., y (c-k+1) (11)

[0094] Then, the vector Y is max-pooled using a max-pooling layer to obtain the character key vector, as shown in formula (12).

[0095] Y' = MaxPooling(Y) (12)

[0096] When k=1, Y' represents the importance of a single character in the sentence and can be regarded as the key value representation information of the sentence.

[0097] In this embodiment, local feature information of sentence vectors is obtained by using convolution and max pooling operations.

[0098] In one embodiment, the steps for obtaining the character vector based on the text are as follows: acquiring the text; inputting the text into a bidirectional gating unit for encoding to obtain the character vector.

[0099] A Bidirectional Gated Recurrent Unit (BiGRU) is a variant of a recurrent neural network that combines the characteristics of gated recurrent units and bidirectional recurrent neural networks. By simultaneously considering forward and backward information over time, BiGRUs can better capture bidirectional dependencies in sequential data. The structure of a BiGRU consists of two independent gated recurrent units, one responsible for forward propagation and the other for backward propagation, and then the outputs of the two independent gated recurrent units are integrated. This structural design allows the model to utilize both past and future information, thus better understanding and predicting the features of sequential data. BiGRUs are commonly used in sequence modeling tasks in natural language processing, such as named entity recognition, sentiment analysis, and machine translation, to more comprehensively capture the semantic and syntactic features of text.

[0100] In this embodiment, after acquiring the text, the text is input into a bidirectional gating unit for encoding to obtain a character vector, which can more comprehensively capture the semantic and syntactic features in the text.

[0101] In one embodiment, the class probability of the text is output based on the feature vector. The specific steps are as follows: perform max pooling on the feature vector to obtain the deep features of the text; and output the class probability of the text based on the deep features of the text.

[0102] For a feature vector BA after feature weighting, it can be partitioned by a convolution kernel of width k as follows: BA = {BA...} (1:k) BA (2:k+1) , ...,BA (c-k+1:c)}, where BA (c-k+1:c)Let S represent the subsequence from the c-k+1th element to the cth element in S. After convolution, the vector T can be obtained, as shown in formula (13).

[0103] T = {t1, t2, ..., t} (c-k+1)} (13)

[0104] In this embodiment, convolution operations are used to mine the feature vectors after feature weighting to obtain deep text features. Then, based on the deep text features, the class probability of the text is output.

[0105] In one embodiment, the class probability of the text is output based on the deep features of the text. The specific steps are as follows: input the deep features of the text into a fully connected layer to obtain the output result; output the class probability of the text based on the output result and the normalized exponential function.

[0106] The deep features are input into the fully connected layer, and the class probability is obtained by taking the index of the maximum value in the array using the normalized exponential function. The calculation method is shown in equations (14) and (15).

[0107] C = MaxPooling(T) (14)

[0108] Output = SoftMax(CW) s +b s (15)

[0109] In equation (15), Output represents the category probability output, the number of dimensions is the total number of data categories, and W s b represents the parameter weight. s This represents the bias parameter.

[0110] In one embodiment, the experimental environment and experimental parameter configuration of this method are shown in Tables 1 and 2.

[0111] Computing resources GPU GTX 1070 operating system Linux Ubuntu 16.04 programming language Python 3.8 Deep learning framework PyTorch 1.9.0

[0112] Table 1 Experimental Environment of the Apparatus

[0113]

[0114]

[0115] Table 2 Experimental parameter configuration

[0116] To verify the effectiveness of the method provided in this application, experiments can be conducted on two datasets: a text dataset from the financial field and a movie review dataset (IMDB).

[0117] To verify whether the method provided in this application can accurately detect and distinguish specific types of text, a text dataset in the financial field can be obtained based on a large number of news texts to evaluate the effectiveness of the method in specific domain text detection. The financial text dataset contains two categories: financial and non-financial, with a total of 20,000 labeled texts. In the experiment, the first 16,000 texts were used as the training set, and the last 4,000 texts as the test set to verify the effectiveness of the method provided in this application on text classification tasks. Relevant information about the financial text dataset is shown in Table 3.

[0118]

[0119] Table 3 Information on text datasets in the financial field

[0120] To verify the effectiveness of the method provided in this application, a comparative experiment can be conducted on an open-source film review dataset. The aforementioned film review dataset is an English film review dataset, containing two categories of data: positive reviews and negative reviews, totaling 50,000 data points, with the training set and test set each comprising half. Relevant information about the film review dataset is shown in Table 4.

[0121]

[0122]

[0123] Table 4. Information from the movie review dataset.

[0124] In the experiment, the category labels of the movie review dataset were marked with 0 and 1, where 0 represents negative emotions and 1 represents positive emotions.

[0125] Because the movie review dataset is well-defined and related works use accuracy for evaluation, this embodiment uses accuracy to evaluate the method's effectiveness on the movie review dataset. However, the text dataset in the financial field has imbalanced class labels, so precision, recall, and the comprehensive evaluation index F1 score (the harmonic mean of precision and recall) are used to evaluate the effectiveness of the method provided in this application. In the experiment, the calculation methods of various evaluation indices are shown in formulas (16) to (19).

[0126] ACC=(TP+TN) / (TP+TN+FP+FN)×100% (16)

[0127] Precision=TP / (TP+FP)×100% (17)

[0128] Recall=TP / (TP+FN)×100% (18)

[0129] F1=(2Precision×Recall) / (Precision+Recall)×100% (19)

[0130] The meaning of the variables in the above formula can be described using the confusion matrix in Table 5.

[0131]

[0132] Table 5 Confusion Matrix

[0133] In Table 5, TP indicates that the model predicted a positive example as a positive example, FN indicates that the model predicted a positive example as a negative example, FP indicates that the model predicted a negative example as a positive example, and TN indicates that the model predicted a negative example as a negative example. Therefore, only TP and TN represent cases where the model predicted correctly.

[0134] To better demonstrate the text classification performance of the method provided in this application, experimental results on relevant datasets are compared. On a movie review dataset, the models compared using the method provided in this application are as follows:

[0135] Tree-Long-Short Term Memory Network (Tree-LSTM): Unlike traditional long short-term memory network models, this model uses a strategy to selectively add multiple forget gates and obtain the feature information that needs attention from the corresponding child nodes. Finally, it uses fully connected layers to obtain the model's classification probability output.

[0136] Deep Convolutional Neural Network (DCNN): This model uses convolutional neural networks as its main units and attempts to extract features using multi-level convolutional operations. Experimental results show that this model has good feature extraction capabilities and achieves good results on a movie review dataset.

[0137] The Convolutional Neural Network (CNN) + Long Short-Term Memory (LSTM) model is used to model text classification tasks. First, the CNN model focuses on local information within the sentence, while the LSTM model focuses on temporal features. Finally, the feature outputs from both models are combined to determine the text type.

[0138] Long Short-Term Memory Network (LSTM): Its basic units are constructed from LSTM network models, using a parallel structure to extract feature information from sentences. Experimental results show that this method achieves good results on text classification tasks.

[0139] Co-Recurrent Neural Network (Co-RNN): A novel architecture based on recurrent neural networks, using a cooperative recurrent neural network model to construct the model. This model effectively leverages the strength of recurrent neural networks in handling time-series information, solving the problem of feature diffusion when processing long texts.

[0140] Unified Recurrent Neural Network (UNICORNN): This model has a similar structure to the cooperative recurrent neural network model. It uses a cascaded recurrent neural network approach to analyze text polarity and explores the advantages of recurrent neural network models in long text modeling tasks.

[0141] Random Temporal Characteristics, Multi-channel Convolutional Neural Network (Rand-TF-MCNN): This model uses a self-attention mechanism to construct word embedding layers and employs multi-channel convolution to focus on local feature information of sentences. Experimental results show that this model has good performance on text classification tasks.

[0142] Collaborative Filtering and Deep Learning Combined Recommendation Model (CFC): This method uses the concept of time series analysis, treating text as a time-series input and modeling it using a deep neural network. Experimental results show that the CFC model can quickly compute text and correctly classify samples.

[0143] Deep Convolutional Neural Network (ALEXNET) model: This model uses the idea of ​​transfer learning, combining multimodal tasks to train the model by converting text data and image data to each other. Experimental results show that using image models for text classification tasks in specific domains can achieve excellent results.

[0144] The accuracy statistics for the movie review dataset are shown in Table 6.

[0145] DCNN 86.80 Tree-LSTM 88.40 Static-BLSTM 86.60 CNN+LSTM 88.90 S-LSTM 87.15 Co-RNN 87.40 UNICORNN 88.40 Rand-TF-MCNN 88.10 CFC 88.40 ALEXNET 87.00 Trans-RCNNs 89.88

[0146] Table 6 Experimental Results

[0147] As shown in Table 6, the method provided in this application has better accuracy than the models listed in the table. In terms of accuracy, the hybrid neural network model (Transformer Region Convolutional Neural Networks, Trans-RCNNs) of this application improves accuracy by approximately 1% compared to the convolutional neural network combined with long short-term memory network model. Simultaneously, compared to other models based on convolutional neural network models or recurrent neural network models, the method provided in this application also shows a certain improvement in accuracy. Compared to deep convolutional neural network models, the method provided in this application has a significant improvement effect, leading by more than three percentage points. Experiments show that the method provided in this application outperforms the convolutional neural network combined with long short-term memory network model in terms of accuracy on the movie review dataset.

[0148] Furthermore, to verify the text classification performance of the method provided in this application in a specific domain, the hybrid neural network model was compared with currently popular neural network models, including: Text-Convolution Neural Network (Text-CNN), Text-Recurrent Neural Network (Text-RNN), Two-way Long Short-Term Memory-attention (BiLSTM-Attention), and Adaptive Large Bidirectional Encoder Representations from Transformers (ALBERT). Experimental results on a text dataset in the financial domain are shown in Table 7.

[0149]

[0150] Table 7 Comparison of experimental results for text datasets in the financial field

[0151] As can be seen from Table 7, the method provided in this application achieves an F1 score of 96% on a text dataset in the financial field without the aid of a pre-trained language model. This is 1.24% higher than the text convolutional neural network model and also shows some improvement compared to the bidirectional long short-term memory neural network-attention model. This further demonstrates the effectiveness of the method provided in this application.

[0152] To investigate the number of model parameters in the method provided in this application, the algorithm model mentioned in the method provided in this application was compared with the model based on the pre-training method. The comparison models are as follows.

[0153] Bidirectional Encoder Base Model (BERT-Base): This method uses a bidirectional encoder model for text classification tasks. Building a model using this method is simple and convenient, and it can achieve good text classification results, but the model has a large number of parameters and high computational cost.

[0154] The BERT-Based ITPT (BERT-based Incrementally Pre-trained Bidirectional Encoder Base Model) is a method that uses a bidirectional encoder model for fine-tuning. It details how parameter fine-tuning experiments are conducted at each layer and proposes a complete solution for classification tasks. Experimental results demonstrate that this method effectively unlocks the potential of pre-trained language models for classification tasks and achieves excellent results.

[0155] Bidirectional encoder large finetune unsupervised data augmentation model (BERT-Large Finetune UDA): This model uses a bidirectional encoder model combined with unsupervised learning to train, in order to solve the problem that the model cannot learn effectively when there are few labeled samples.

[0156] Large Cross-Attention Network (XLNET-Large): This model uses an autoregressive method to train a pre-trained language model and then uses this model for parameter fine-tuning on downstream tasks. Experimental results show that the Large Cross-Attention Network performs well on text classification tasks and achieves better results than bidirectional encoder models on some datasets.

[0157] BERT-Base 110M BERT-Base ITPT 110M BERT-Large Finetune UDA 330M XLNET-Large 340M Trans-RCNNs 18M

[0158] Table 8 Comparison of Model Parameter Quantities

[0159] As shown in Table 8, the model involved in the method provided in this application has fewer parameters than other models. Compared with large-scale pre-trained models, the model involved in the method provided in this application has the characteristics of low time consumption and fast computation, while supporting online deployment requirements in natural scenarios and meeting the system's computational needs.

[0160] Experimental results show that, without using a large-scale pre-trained language model, the method provided in this application achieves state-of-the-art text classification results on text datasets in the financial domain and movie review datasets. On the text dataset in the financial domain, the F1 score is 96.46%, which is 0.61% higher than the adaptive large bidirectional encoder model; on the movie review dataset, the accuracy is 89.88%, which is 2.88% higher than the deep convolutional neural network model.

[0161] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0162] Based on the same inventive concept, this application also provides a financial text classification device for implementing the aforementioned financial text classification method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the financial text classification device provided below can be found in the limitations of the financial text classification method described above, and will not be repeated here.

[0163] In one exemplary embodiment, such as Figure 6 As shown, a text classification device for the financial field is provided, wherein:

[0164] The text acquisition module 601 is used to acquire the text to be classified;

[0165] The category probability prediction module 602 is used to input the text into a pre-built hybrid neural network model, so that the hybrid neural network model performs the following processing steps: through the model encoding layer, the text is converted into a sentence vector; through the feature attention layer, the sentence vector is convolved to obtain a character key vector and a character vector is obtained based on the text; the character key vector and the character vector are weighted to obtain a feature weight vector; through the text classification layer, the feature weight vector is convolved to obtain a feature vector, and the category probability of the text is output based on the feature vector;

[0166] The category probability acquisition module 603 is used to acquire the category probability of the text;

[0167] The financial field text judgment module 604 is used to determine whether the text belongs to the financial field based on the category probability of the text.

[0168] In one embodiment, the category probability prediction module 602 is further configured to: encode the meaning of words in the text through a word embedding layer; represent the position of the words through a position encoding layer to obtain the text in vector form; and input the text in vector form into a pre-built hybrid neural network model.

[0169] In one embodiment, the model encoding layer includes a multi-head attention mechanism and a forward propagation layer. The category probability prediction module 602 is further configured to: input the text into the multi-head attention mechanism to obtain a concatenated vector; and input the concatenated vector into the forward propagation layer for nonlinear transformation to obtain a sentence vector.

[0170] In one embodiment, the category probability prediction module 602 is further configured to: perform a convolution operation on the sentence vector to obtain a grammatical feature information vector; and perform a max pooling operation on the grammatical feature information vector to obtain the character key vector.

[0171] In one embodiment, the category probability prediction module 602 is further configured to: acquire the text; input the text into a bidirectional gating unit for encoding to obtain a character vector.

[0172] In one embodiment, the category probability prediction module 602 is further configured to: perform max pooling on the feature vector to obtain the deep features of the text; and output the category probability of the text based on the deep features of the text.

[0173] In one embodiment, the category probability prediction module 602 is further configured to: input the deep features of the text into a fully connected layer to obtain an output result; and output the category probability of the text based on the output result and the normalized exponential function.

[0174] The modules in the aforementioned text classification device for the financial field can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0175] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data for a text classification method in the financial field. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a text classification method in the financial field.

[0176] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A text classification method in the financial field, characterized in that, The method includes: Get the text to be categorized; The text is input into a pre-built hybrid neural network model, which then performs the following processing steps: The text is converted into a sentence vector through a model encoding layer; a feature attention layer is used to convolve the sentence vector to obtain a character key vector and, based on the text, a character vector is obtained; the character key vector and the character vector are weighted together to obtain a feature weight vector; a text classification layer is used to convolve the feature weight vector to obtain a feature vector, and, based on the feature vector, the class probability of the text is output; the weighting operation involves performing a dot product operation between the character key vector and the character vector. The step of inputting the text into a pre-built hybrid neural network model includes: encoding the meaning of words in the text through a word embedding layer; representing the position of the words through a position encoding layer to obtain the text in vector form; and inputting the text in vector form into the pre-built hybrid neural network model. The model encoding layer includes a multi-head attention mechanism and a forward propagation layer. The process of converting the text into a sentence vector through the model encoding layer includes: inputting the text into the multi-head attention mechanism to obtain a concatenated vector; inputting the concatenated vector into the forward propagation layer for nonlinear transformation to obtain a sentence vector; wherein, the output vectors of multiple self-attention mechanisms are concatenated to obtain the concatenated vector. Obtain the category probability of the text; Based on the category probability of the text, determine whether the text belongs to the financial field.

2. The method according to claim 1, characterized in that, The process of performing a convolution operation on the sentence vector to obtain a character key vector includes: Perform a convolution operation on the sentence vector to obtain a grammatical feature information vector; The character key vector is obtained by performing max pooling on the grammatical feature information vector.

3. The method according to claim 1, characterized in that, The process of obtaining a character vector based on the text includes: Obtain the text; The text is input into a bidirectional gating unit for encoding to obtain a character vector.

4. The method according to claim 1, characterized in that, The step of outputting the category probability of the text based on the feature vector includes: Max pooling is performed on the feature vector to obtain the deep features of the text; Based on the deep features of the text, output the category probability of the text.

5. The method according to claim 4, characterized in that, The step of outputting the category probability of the text based on its deep-level features includes: The deep features of the text are input into a fully connected layer to obtain the output result; Based on the output and the normalized exponential function, the category probability of the text is output.

6. A text classification device for the financial field, characterized in that, The device includes: The text acquisition module is used to acquire the text to be classified. The category probability prediction module is used to input the text into a pre-built hybrid neural network model, so that the hybrid neural network model performs the following processing steps: Through the model encoding layer, the text is converted into a sentence vector; through the feature attention layer, the sentence vector is convolved to obtain a character key vector, and a character vector is obtained based on the text; the character key vector and the character vector are weighted to obtain a feature weighted vector; through the text classification layer, the feature weighted vector is convolved to obtain a feature vector, and the category probability of the text is output based on the feature vector; the category probability acquisition module is used to acquire the category probability of the text; the weighting operation is to perform a dot product operation between the character key vector and the character vector. The category probability prediction module is also used to encode the meaning of words in the text through a word embedding layer; represent the position of the words through a position encoding layer to obtain the text in vector form; and input the text in vector form into a pre-built hybrid neural network model; The category probability prediction module is also used to input the text into a multi-head attention mechanism to obtain a concatenated vector; input the concatenated vector into a forward propagation layer for nonlinear transformation to obtain a sentence vector; wherein, the output vectors of multiple self-attention mechanisms are concatenated to obtain the concatenated vector; The financial text judgment module is used to determine whether the text belongs to the financial field based on the category probability of the text.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images