Real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU

Abstract

The invention discloses a real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU, and the model comprises a high-target BERT module which generates semantic-rich sentence representation for a specific target through BERT, and enhances the correlation through an attention mechanism; a random masking GNN module, wherein the random masking GNN module clears the relationship between the data through edges and nodes of a random masking graph; and an attention GRU module, wherein the attention GRU module is used for capturing the interaction between the new data and the historical data. According to the real estate enterprise risk prediction method and system, the real estate enterprise risk prediction method has important application value in practical application, an accurate and reliable solution can be provided for a real estate enterprise risk prediction task, and the risk prediction effect is helped to be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise risk prediction, and in particular to a real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU. Background Art

[0002] The field of stock series forecasting has attracted significant research interest due to the high accuracy and effectiveness of deep learning techniques in capturing complex patterns in financial data. Techniques such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and gated recurrent units (GRUs) are widely used to analyze stock data and predict future price trends. Furthermore, pre-trained models such as BERT are widely used to extract relevant features from stock-related news, thereby enhancing the predictive power of financial forecasting models.

[0003] However, existing methods face challenges in analyzing multimodal data (including news, policies, and stock information) for comprehensive risk assessment, often overlooking the interactions between different data modalities. Some models rely solely on stock data, ignoring the impact of corporate news and national policies on stock price fluctuations. Other models, while integrating news data, fail to fully exploit the connection between news and stock data to improve prediction accuracy. Furthermore, many aspect-based semantic representation models for news text processing lack attention to target information, reducing their effectiveness in downstream tasks.

[0004] Technical terms: GNN: Graph Neural Network; GRU: Gated Recurrent Unit; HTBERT: High-Target BERT; RMGNN: Random Masked GNN; AGRU: Attention GRU; Enterprise Risk Index Prediction Model: HRAGRU Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a real estate enterprise risk prediction model based on high-target semantics BERT and improved GRU, which has important application value in the practical application of real estate enterprise risk prediction methods, can provide accurate and reliable solutions for real estate enterprise risk prediction tasks, and help improve the effect of risk prediction. In order to achieve the above-mentioned purpose and other advantages according to the present invention, a real estate enterprise risk prediction model based on high-target semantics BERT and improved GRU is provided, including:

[0006] A high-target BERT module, which generates semantically rich sentence representations for a specific target through BERT and enhances their relevance through an attention mechanism;

[0007] A random masking GNN module that clarifies the relationship between data by randomly masking the edges and nodes of the graph;

[0008] Attention GRU module, the attention GRU module is used to capture the interaction between new data and historical data.

[0009] Preferably, the high-target BERT module extracts semantic information from each item through a pre-trained BERT model, and specifically obtains deeper language knowledge through a semantic sequence layer. After applying a Transformer encoder to each hidden layer, a softmax layer is added to integrate global features.

[0010] Preferably, the random masked GNN module preprocesses news data by merging multiple entries into one representation, specifically:

[0011] Calculate the average representation of news features on day t;

[0012] The graph consists of a maximum of five nodes and forms a fully connected structure at each time step t; random masking is applied to the edges and nodes to generate m' subgraphs. And m' masking maps are used in the feature extraction process of the GNN encoder to calculate the encoder of each node;

[0013] The integrated data is obtained by stitching together and The layered diagram representation is obtained.

[0014] Preferably, the attention GRU module is an improved gated recurrent unit model, which introduces an attention mechanism in the GRU gating process, enhances the model's ability to capture the association between history and new data, and evaluates the correlation between input data and the previous hidden state through the attention mechanism, thereby improving the accuracy of the update gate calculation.

[0015] Compared with the existing technology, the present invention has the following advantages: BERT is used to generate semantically rich sentence representations for specific targets through HTBERT, and its relevance is enhanced through the attention mechanism. Secondly, RMGNN clarifies the relationship between data by randomly masking the edges and nodes of the graph, effectively alleviating the problem of node scarcity. Finally, AGRU introduces an attention mechanism in the GRU gating process, enhancing the model's ability to capture the association between historical and new data. The real estate enterprise risk prediction method proposed in this invention has important application value in practical applications. It can provide an accurate and reliable solution for the real estate enterprise risk prediction task, helping to improve the effectiveness of risk prediction. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0017] For each stock, given an input sequence X = {x t-T+1 ,x t-T+2 ,…,x t}, where T is the sequence length, Indicates five data categories. and Represent the daily stock data, current policy data, national macroeconomic data and corporate financial data of day t respectively. In addition, is the j-th news data on the t-th day. sd, pd, md and fd represent and The present invention aims to predict the enterprise risk sequence of the next T' trading days, which is expressed as Y = {y t+1 ,y t+2 ,…,y t+T '}, where y t represents the real estate enterprise risk index on day t. The calculation formula of this index is where close t is the 5-day exponential moving average of the closing price on day t. The function rank ranks the stock volatility of each real estate company, and the result is expressed as a percentage. t The range is 0 to 1, with values greater than 0.5 indicating higher risk and values less than 0.5 indicating lower risk.

[0018] The real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU includes:

[0019] The high-target BERT module generates semantically rich sentence representations for specific targets through BERT and enhances their relevance through the attention mechanism. The present invention uses the pre-trained BERT model to extract semantic information from each entry. Formally, the original news sentence is represented as ST = {[CLS], w1,…, t1,…, t m ,…,w n ,[SEP]}, where ST contains n words, each w i Represents a word in ST. Target enterprise TE={t1,…,t m} consists of m words, where each t iis a specific word in ST that belongs to TE. Subsequently, the present invention introduces explicit marking <asp>and< / asp> To represent aspect boundaries: ST'={[CLS],w1,…, <asp>,t1,…,t m ,< / asp>,…,w n ,[SEP]}.

[0020] This paper uses semantic sequence layers to obtain deeper language knowledge. After applying the Transformer encoder to each hidden layer, a softmax layer is added to integrate global features. The calculation formula is Where W s ∈R d×b and b s ∈R d is a trainable parameter. Here, o i ∈R b is the output of the i-th word output by the encoder, b is the output dimension of the encoder, and d is the dimension of the semantic representation vector input to the attention layer.

[0021] The target entity is In the representation, denoted as It is also defined by the following formula: where o' i ∈R b Represents the feature representation output of the i-th word, W t ∈R d×b and b t ∈R d is a trainable parameter.

[0022] get and Then, the present invention applies attention mechanism to calculate news data and the normalized scores of the target entity, respectively and Finally, semantic dense representation The formula is Where W q 、W k and W v ∈R a×d is a trainable Transformer attention parameter, a is the dimension of the semantic representation vector from the attention layer. ⊙ denotes the Hadamard product, exp denotes the exponential function with the natural constant e as the base, and · denotes scalar multiplication with a matrix.

[0023] The random masking GNN module clarifies the relationship between data by randomly masking the edges and nodes of the graph. In most cases, the number of daily news items for real estate companies far exceeds that of other types of data. Therefore, the present invention preprocesses news data by combining multiple items into a single representation. First, the present invention calculates the average representation c of the news features on day t. t ∈R a , and its calculation formula is Final feature representation The calculation is as follows:

[0024]

[0025] Where n' represents the total number of news articles on day t. Q 、W K 、W V ∈R a×a is a trainable parameter. || represents vector concatenation.

[0026] The graph consists of at most five nodes And form a fully connected structure at each time step t, denoted as Apply random masking to edges and nodes to generate m' subgraphs, Original image And m' mask maps are used in the feature extraction process of the GNN encoder. The encoder calculation formula for each node is as follows:

[0027]

[0028]

[0029] Among them, γ i represents the node category, σ is the activation function, q T ∈R O , b g ∈R O are all trainable parameters. O and P are the dimensions of the attention vector and graph node respectively. and are trainable parameters for node categories i and j. i and η j is the node vector, N i is the neighbor set of node i, o ij is the attention value, η' i ∈R P It is the new node vector after GNN transformation.

[0030] The final integrated data is obtained by splicing and The layered graph representation is obtained, and its calculation formula is

[0031]

[0032] Among them, d t Represents the final representation of stock-related data on day t. Function Num calculation graph The total number of nodes in the.

[0033] The attention GRU module is used to capture the interaction between new data and historical data. Traditional GRUs calculate the update gate through linear transformations and activation functions, ignoring the dynamic correlation between input data and hidden states. This invention enhances the GRU and uses an attention mechanism to evaluate the correlation between input data and the previous hidden state, thereby improving the accuracy of the update gate calculation. The calculation formulas for the reset gate, update gate, new memory content, hidden state, and output are as follows:

[0034]

[0035] r t =σ(U r ⊙x' t +W r ⊙h t-1 +b r )

[0036] z t =σ(Att(Tanh(W x x' t +b x })||Tanh(W' h h t-1 +b h ))+b z )

[0037]

[0038] Among them, x' t ∈R I , h t-1 , h t ∈R H are vectors representing the input, previous hidden state, and current hidden state respectively. σ is the activation function sigmoid. t Represents the internal reset vector, z t Represents the update vector. U r ∈R I , W r ∈R H , b r , U h ∈R H×I , W h ∈R H×H , are the trainable parameters corresponding to the reset gate and the new memory content. I = 2a + m' × P + sd + pd + fd + md and H denote the dimensions of the input and hidden layers, respectively. W represents a trainable attention function, where D represents the dimension of the attention vector. x ∈R D×I , W' h ∈R D×H , b x ∈R D , b h ∈R D and b z is a trainable parameter.

[0039] To ensure the interpretability of the proposed HRAGRU model, we design a unique loss function for each module. The loss function can be calculated as follows:

[0040]

[0041] Among them, S t Represents the set of all news items on day t, from which the function Num' is used to retrieve individual news items. l' t and y' t Through multi-layer perceptron (MLP) from d t , and h t Obtained in. and They represent the predicted sentiment polarity and corresponding sentiment label of the j-th news item on the t-th day, with positive and negative sentiment labels being 1 and 0 respectively. Similarly, l' t and l t Represent the predicted and actual real estate enterprise risk index on day t, respectively, and are derived based on the graph structure representation. t and y t denote the predicted and actual values of the real estate enterprise risk index on day t using the time series model. Data from day 1 to day T are used to predict the risk index from day T+1 to day T+T'. λ1, λ2, and λ3 adjust the weights of different loss functions.

[0042] In summary, the present invention proposes HRAGRU, a new enterprise risk index prediction model, which includes High Target BERT (HTBERT), Random Masked GNN (RMGNN) and Attention GRU (AGRU). First, HTBERT uses BERT to generate semantically rich sentence representations for specific targets and enhances their relevance through the attention mechanism. Secondly, RMGNN clarifies the relationship between data by randomly masking the edges and nodes of the graph, effectively alleviating the node scarcity problem. Finally, AGRU introduces the attention mechanism in the GRU gating process, which enhances the model's ability to capture the correlation between historical and new data. The real estate enterprise risk prediction method proposed in the present invention has important application value in practical applications. It can provide accurate and reliable solutions for real estate enterprise risk prediction tasks and help improve the effect of risk prediction.

[0043] The number of devices and processing scales described herein are intended to simplify the description of the present invention, and the application, modification, and variation of the present invention will be apparent to those skilled in the art. Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiment. They can be applied to various fields suitable for the present invention. For those skilled in the art, additional modifications can be easily implemented. Therefore, the present invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the claims and their equivalents.< / asp>

Claims

1. A real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU, characterized by: include: A high-target BERT module, which generates semantically rich sentence representations for a specific target through BERT and enhances their relevance through an attention mechanism; A random masking GNN module that clarifies the relationship between data by randomly masking the edges and nodes of the graph; Attention GRU module, the attention GRU module is used to capture the interaction between new data and historical data.

2. The real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU according to claim 1 is characterized in that: The high-target BERT module extracts semantic information from each item through the pre-trained BERT model, and specifically obtains deeper language knowledge through the semantic sequence layer. After applying the Transformer encoder to each hidden layer, a softmax layer is added to integrate global features.

3. The real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU according to claim 1 is characterized in that: The random masked GNN module preprocesses news data by merging multiple entries into one representation, specifically: Calculate the average representation of news features on day t; The graph consists of a maximum of five nodes and forms a fully connected structure at each time step t; random masking is applied to the edges and nodes to generate m' subgraphs. And m' masking maps are used in the feature extraction process of the GNN encoder to calculate the encoder of each node; The integrated data is obtained by stitching together and The layered diagram representation is obtained.

4. The real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU according to claim 1 is characterized in that: The attention GRU module is an improved gated recurrent unit model that introduces an attention mechanism into the GRU gating process, enhancing the model's ability to capture the association between historical and new data. The attention mechanism is used to evaluate the correlation between input data and the previous hidden state, thereby improving the accuracy of update gate calculations.

5. The real estate enterprise risk prediction model based on high-target semantic BERT and improved GRU according to claim 1 is characterized in that: Including the loss function, the loss function can be calculated as follows: Among them, S t Represents the set of all news items on day t, from which the function Num' is used to retrieve individual news items. l' t and y' t Through multi-layer perceptron (MLP) from d t , and h t Obtained in. and They represent the predicted sentiment polarity and corresponding sentiment label of the j-th news item on the t-th day, with positive and negative sentiment labels being 1 and 0 respectively; similarly, l' t and l t Represent the predicted and actual real estate enterprise risk indexes on day t, respectively, which are derived based on the graph structure representation; y' t and y t They represent the predicted value and actual value of the real estate enterprise risk index on day t by the time series model; the data from day 1 to day T are used to predict the risk index from day T+1 to day T+T'; λ1, λ2, and λ3 adjust the weights of different loss functions.

Images