Urban real estate price index prediction method and device based on online news sentiment analysis

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an urban real estate price index prediction method and device based on online news sentiment analysis. The method comprises the following steps: step A, collecting news text data according to keywords, preprocessing the news text data, and generating a standard real estate online news data set by applying a structured method; step B, calculating a news emotion value corresponding to each piece of news in the real estate online news data set based on a BERT emotion analysis framework, and calculating a real estate emotion index based on the news emotion values; and step C, obtaining a historical real estate price index, forming a real estate index with the real estate emotion index, and inputting the historical real estate index into the VAR model to obtain a real estate price index of the next period. According to the method, a new real estate emotion RES index is designed, the vector autoregression model of the RES index and the real estate price index is constructed, and high-precision prediction of the real estate price index can be realized.

Description

Technical Field

[0005] The present invention relates to the field of artificial intelligence technology, and in particular, to a method and device for predicting urban real estate price index based on online news sentiment analysis. Background Art

[0006] The real estate price index is a key indicator for evaluating the operation status of the real estate market. Generally, it is regularly released by the national authoritative department, providing a direct description of the price change trend of the real estate market for stakeholders in the real estate industry. However, due to the complexity of the economic statistical process, the release of the real estate price index is always significantly lagged. Moreover, due to the uniqueness of the real estate market and the diversity of statistical data sources, different countries adopt different methods to construct the real estate price index, and there is no unified calculation standard, which also brings difficulties to the prediction and evaluation of the real estate price index.

[0007] At the same time, the rapid development of information technology has made the Internet the main medium for news dissemination and sharing, providing a new way to perceive market dynamics. By deeply analyzing the rich and diverse news content on the Internet, researchers can quickly and comprehensively capture the public's emotions towards market fluctuations, and then timely insight into the development trend of the industry. There have been many studies using natural language processing technology and deep learning technology to deeply understand the emotional semantics of massive news, but the existing methods have certain limitations in terms of data sources, technical methods, etc., and cannot provide a high-precision and feasible application approach for real estate market trend prediction. Summary of the Invention

[0008] In order to solve the problem that the existing real estate prediction methods have limitations in data sources and technical methods, resulting in low accuracy of real estate market trend prediction, the present invention provides a method and device for predicting urban real estate price index based on online news sentiment analysis. By designing a new real estate sentiment RES index and constructing a vector autoregressive model of the RES index and the real estate price index, high-precision prediction of the real estate price index can be achieved.

[0009] In the first aspect, a method for predicting urban real estate price index based on online news sentiment analysis provided by the present invention includes:

[0010] Step A: Collect news text data according to keywords, preprocess the news text data, and generate a standardized real estate online news dataset by using a structured method;

[0011] Step B: Calculate the news sentiment value corresponding to each news in the real estate online news dataset based on the sentiment analysis framework of BERT, and calculate the real estate sentiment index based on the news sentiment value; including:

[0012] Tokenize the news texts in the real estate news dataset, and use the BERT pre-trained model to analyze the tokenization results to obtain the probabilities of each sentiment category to which the news texts belong. The sentiment type with the highest probability value is the news sentiment value. Summarize the news sentiment values by time to obtain the real estate sentiment index; the BERT pre-trained model includes a word vector embedding layer, a bidirectional Transformer encoder, a Linear layer, and a Softmax layer;

[0013] Step C: Obtain the historical real estate price index, and form a real estate index with the real estate sentiment index. Input the historical real estate index into the VAR model to obtain the real estate price index for the next period.

[0014] Further, the preprocessing includes data denoising, format unification, and elimination of invalid information.

[0015] Further, the keywords include a time range, a city name, and real estate.

[0016] Further, tokenizing the news texts in the real estate news dataset specifically includes:

[0017] Use the WordPiece algorithm to tokenize the news text X in the real estate news dataset into W = <CLS, w 1 , w 2 , …, SEP, w i , …, SEP>, where w i represents the i-th word in the news text, CLS represents the start token of the sentence, and SEP represents the separator token between sentences.

[0018] Further, the training process of the BERT pre-trained model includes:

[0019] Construct a BERT pre-trained model. Use a BERT model pre-trained with 250 million Chinese Wikipedia vocabulary data. The BERT model has 12 layers of bidirectional Transformer encoders, with 12 self-attention heads in each layer, and add the Linear layer and the Softmax layer after the bidirectional Transformer encoder;

[0020] Fine-tune the BERT pre-trained model. Use the labeled news text dataset to train the BERT pre-trained model, and use cross-entropy loss as the loss function.

[0021] Further, the summarizing the news sentiment values by time to obtain the real estate sentiment index specifically includes: Summarize the news sentiment values predicted by the BERT pre-trained model monthly, and calculate the real estate sentiment index through the following formula:

[0022]

[0023] Among them, d.label represents the news sentiment value corresponding to the news text, n represents the number of news texts in the news text dataset of the current month, r represents the real estate sentiment index, and timespan represents the time range of the current month.

[0024] Furthermore, the expression formula of the VAR model is as follows:

[0025]

[0026] Among them, represents the real estate index in the t time period, c t represents the real estate price index in the t time period, r t represents the real estate sentiment index in the t time period, A 0 =[a c0 , a h0 ′ represents the constant vector, represents the parameter matrix, k represents the lag order and k≥i>1, ε t =[ε ct , ε ht ′ represents the residual vector.

[0027] Furthermore, the ADF method is used to detect the stationarity of the predicted real estate sentiment index and real estate price index. If any index of the VAR model is non-stationary, the real estate index is subjected to differencing processing, and the processing formula is as follows:

[0028]

[0029] Among them, represents the real estate index in the t-1 time period.

[0030] Furthermore, the Akaike information criterion is used to solve the optimal lag order k of the VAR model, and the calculation formula is as follows:

[0031] AIC(k)=-2*ln(L)+2*k

[0032] Among them, L represents the likelihood value of the VAR model;

[0033] Based on the determined lag order k, the least squares method is used to estimate the parameters A 0 , A i and ε t , and the calculation formula is as follows:

[0034]

[0035] Among them, A=[A0 , A 1 , …, A k represents a set of parameter matrices, and Y = [Y t-1 , …, Y t-k represents the independent variable matrix;

[0036] After obtaining A, substitute it into the VAR model to solve for the residual vector ε t .

[0037] In a second aspect, a device for predicting a city real estate price index based on online news sentiment analysis provided by the present invention includes:

[0038] A data acquisition module, configured to collect news text data according to keywords, preprocess the news text data, and generate a standardized real estate online news data set by using a structured method;

[0039] An emotion index prediction module, configured to analyze the news sentiment value corresponding to each news in the real estate online news data set based on a sentiment analysis framework of BERT, and calculate a real estate emotion index based on the news sentiment value; including:

[0040] Segment the news text in the real estate news data set, use the BERT pre-trained model to analyze the segmented results to obtain the probabilities of each sentiment category to which the news text belongs, the sentiment type with the highest probability value is the news sentiment value, and summarize the news sentiment values over time to obtain the real estate emotion index; the BERT pre-trained model includes a word vector embedding layer, a bidirectional Transformer encoder, a Linear layer, and a Softmax layer;

[0041] A housing price index prediction module, configured to obtain a historical real estate price index, form a real estate index with the real estate emotion index, and input the historical real estate index into the VAR model to obtain the real estate price index for the next period.

[0042] Advantages of the present invention:

[0043] First, in order to accurately capture the public's emotional perception of the real estate market, the present invention constructs a sentiment analysis model based on BERT, analyzes the emotions of online news through Chinese sentiment analysis technology, and thus obtains the Real Estate Sentiment (RES) index. Subsequently, a VAR model of the RES index and the official real estate price index is further constructed to comprehensively and deeply explore the interaction mechanism between the real estate sentiment index and the real estate price index, and can accurately predict the real estate price index and the real estate sentiment index in the next period. And the new housing price index and the second-hand housing price index can be obtained, and thus VAR models of the new housing price index and the second-hand housing price index and the RES index are respectively constructed, and the new housing price index and the second-hand housing price index in the next period can be respectively predicted.

[0044] This method has been verified in four cities of Beijing, Shanghai, Guangzhou and Shenzhen. Using the online news data from 2011 to 2022 for 12 years, sentiment analysis is carried out, and the RES index of the target city is obtained. Then, 8 prediction models are constructed by using the RES index and the new housing price index and the second-hand housing price index of the four cities respectively to realize the prediction of the new housing and second-hand housing price indexes in the next month. The average value of the root mean square error RMSE of the prediction results is 0.161, showing good prediction performance overall. Description of the Drawings

[0045] Figure 1 It is a schematic flow chart of a method for predicting the urban real estate price index based on online news sentiment analysis provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic framework diagram of a method for predicting the urban real estate price index based on online news sentiment analysis provided by an embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of the sentiment analysis framework based on BERT provided by an embodiment of the present invention. Detailed Embodiments

[0048] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0049] As Figure 1 and Figure 2 shown, an embodiment of the present invention provides a method for predicting the urban real estate price index based on online news sentiment analysis, including:

[0050] Step A: Data collection;

[0051] Collect news text data according to keywords, preprocess the news text data, and generate a standardized real estate online news dataset using a structured method;

[0052] Specifically, first, search keywords such as time range, city name, and real estate are used to crawl online news text data closely related to real estate from news platforms. Then, preprocess the news text data, mainly including operations such as data denoising, format unification, and elimination of invalid information. Subsequently, a standardized real estate online news dataset is generated using a structured method, denoted as where city is the city name. In dataset D city the i-th news record is represented as X is the news text, and date is the news release date.

[0053] Step B: News semantic analysis, calculate the news sentiment value corresponding to each news in the real estate online news dataset based on the sentiment analysis framework of BERT, and calculate the real estate sentiment index based on the news sentiment value; including:

[0054] As Figure 3 shown, tokenize the news text in the real estate news dataset, use the BERT pre-trained model to analyze the tokenization results to obtain the probabilities of each sentiment category to which the news text belongs, and the sentiment type with the highest probability value is the news sentiment value. Summarize the news sentiment values by time to obtain the real estate sentiment index; among them, the BERT pre-trained model includes a word vector embedding layer, a bidirectional Transformer encoder, a Linear layer, and a Softmax layer.

[0055] Specifically, tokenize the news text in the real estate news dataset, specifically including:

[0056] Use the WordPiece algorithm to tokenize the news text X in the real estate news dataset into W = <CLS, w 1 , w 2 , …, SEP, w i , …, SEP>, where w i represents the i-th word in the news text, CLS represents the start token of the sentence, and SEP represents the separator token between sentences.

[0057] To fully understand the prediction of the news text, the BERT pre-trained model combines three embedding forms of word vectors, sentence vectors, and position vectors as the input vector of the bidirectional Transformer encoder, that is, E = <E i >, where the comprehensive embedding of the i-th word is denoted as E i .

[0058] The training process of the BERT pre-trained model includes:

[0059] Construct the BERT pre-trained model. First, use the BERT model pre-trained with 250 million Chinese Wikipedia vocabulary data. The BERT model has 12 layers of bidirectional Transformer encoders, each layer is equipped with 12 self-attention heads, and a Linear layer and a Softmax layer are added after the bidirectional Transformer encoder;

[0060] Fine-tune the BERT pre-trained model. Use the labeled news text dataset to train the BERT pre-trained model, and use the cross-entropy loss as the loss function.

[0061] Specifically, the labeled news text dataset is denoted as where city is the name of the city. In the dataset D city the i-th news record is represented as X is the news text, date is the news release date, labe represents the sentiment tendency of the news text, which is divided into "positive" (labe = 1), "neutral" (labe = 0), and "negative" (labe = -1). Use the labeled news text dataset to adjust the pre-trained parameters of the BERT model. In this process, the BERT model is used to extract the high-dimensional embedding features T of the news text. Subsequently, a Linear layer converts the high-dimensional embedding into the category dimension of the news semantics, that is, classify for the three sentiment tendencies (categories). The conversion formula of the Linear layer is O = W·T + b, where O represents the output of the linear layer, W is the weight, and b is the bias. Then the Softmax layer normalizes O to output the sentiment category probability of the text. In addition, in the fine-tuning stage, we use the cross-entropy loss as the loss function, that is where L represents the loss value, y i represents the predicted value, represents the true value. At the same time, adjust parameters such as the batch size, learning rate, and number of training epochs. Finally, the trained BERT model can be used to analyze the news text in the real estate online news dataset in step A, determine the sentiment category label with the highest probability in the classification layer, and output the news sentiment value corresponding to the news text data.

[0062] Specifically, summarize the news sentiment values predicted by the BERT pre-trained model monthly, and calculate the real estate sentiment index through the following formula:

[0063]

[0064] Among them, d.label represents the news sentiment value corresponding to the news text, n represents the number of news texts in the news text dataset of the current month, r represents the real estate sentiment index, and timespan represents the time range of the current month.

[0065] Step C: Predict the real estate price index.

[0066] Obtain the historical real estate price index, and form a real estate index with the real estate sentiment index. Input the historical real estate index into the VAR model to obtain the real estate price index for the next period.

[0067] Specifically, collect the real estate price indices of each city in the historical period. The VAR model is used to evaluate the quantitative relationship between the real estate sentiment index and the real estate price index. For the k-order VAR model of R and C, the formula is as follows:

[0068]

[0069] Among them, represents the real estate index at time t, c t represents the real estate price index at time t, r t represents the real estate sentiment index at time t, A 0 =[a c0 ,a h0 ′ represents the constant vector, represents the parameter matrix, k represents the lag order and k≥i>1, ε t =[ε ct ,ε ht ′ represents the residual vector.

[0070] Furthermore, since the VAR model must be constructed on a stationary time series or show a cointegration relationship on a non-stationary time series, that is, the modulus of the model eigenvalue must be less than 1 (i.e., the eigenvalue is inside the unit circle). The ADF method is used to detect the stationarity of the predicted real estate sentiment index and real estate price index. If any index of the VAR model is non-stationary, the real estate index is differenced, and the processing formula is as follows:

[0071]

[0072] Among them, represents the real estate index at time t-1.

[0073] It can be understood that most application scenarios of the VAR model are based on the assumption of stationary data, which helps to ensure the consistency and reliability of model parameter estimation, and at the same time can avoid the problem of spurious regression caused by non-stationary data. Although the cointegrated VAR can handle non-stationary data, it has limitations, such as the complex cointegration relationship test and unstable results, and it is not as intuitive and effective as the VAR model based on stationary data when explaining short-term dynamic adjustment. Therefore, the VAR model of the optimal embodiment of the present invention is constructed on a stationary time series.

[0074] Furthermore, the Akaike information criterion is used to solve the optimal lag order k of the VAR model, and the calculation formula is as follows:

[0075] AIC(k) = -2 * ln(L) + 2 * k

[0076] where L represents the likelihood value of the VAR model;

[0077] Specifically, AIC is an index to measure the goodness of fit of a statistical model, and the smaller the value, the better the model. Traverse different k values and calculate AIC(k) of the VAR model, and determine the optimal lag order by selecting the smallest value.

[0078] Based on the determined lag order k, the least squares method is used to estimate the parameters A 0 , A i and ε t of the VAR model, and the calculation formula is as follows:

[0079]

[0080] where A = [A 0 , A 1 , …, A k represents the parameter matrix set, Y = [Y t-1 , …, Y t-k represents the independent variable matrix; after obtaining A, substitute it into the VAR model to solve the residual vector ε t .

[0081] Specifically, based on the determined lag order k, we estimate the parameters of the VAR model, that is, A 0 , A i and ε t in formula 1. Since C and R are stationary time series, we can assume that their means and variances are close to 0. According to the Gauss Markov theorem: the regression coefficient estimation obtained by ordinary least squares (OLS) is optimal.

[0082] Based on the above embodiments, this embodiment further provides a method for predicting the new housing price index and the second-hand housing price index. It collects the new housing price index Pn = {pn date} and the time series data Ps = {ps date} of the second-hand housing price index in the same historical period for each city. Where pn date and ps date respectively represent the new housing price index and the second-hand housing price index on a given date date. And a VAR model for new housing and a VAR model for second-hand housing are respectively established to predict the price indices of new housing and second-hand housing.

[0083] This invention takes Chinese online real estate news data as the research object and proposes a housing price index prediction scheme based on a large amount of network news data, aiming to achieve real-time and accurate prediction of the real estate price index in Chinese cities and open up a new path for deeply understanding and analyzing the trends of the Chinese real estate market. First, in order to accurately capture the public's emotional perception of the real estate market, this invention constructs a sentiment analysis model based on BERT, analyzes the emotions of online news through Chinese sentiment analysis technology, and constructs a real estate sentiment (RES). Subsequently, a VAR model of the RES index and the official new housing and second-hand housing price indices is further constructed to comprehensively and deeply explore the interaction mechanism between the sentiment index and the housing price index.

[0084] This embodiment of the invention further provides a device for predicting the urban real estate price index based on online news sentiment analysis, including:

[0085] A data collection module, which is used to collect news text data according to keywords, preprocess the news text data, and generate a standardized real estate online news data set by using a structured method;

[0086] A sentiment index prediction module, which is used to analyze the news sentiment value corresponding to each news in the real estate online news data set based on the sentiment analysis framework of BERT, and calculate the real estate sentiment index based on the news sentiment value; including:

[0087] Segment the news text in the real estate news data set, use the BERT pre-trained model to analyze the segmented results to obtain the probabilities of each sentiment category to which the news text belongs, and the sentiment type with the highest probability value is the news sentiment value. Summarize the news sentiment values over time to obtain the real estate sentiment index; the BERT pre-trained model includes a word vector embedding layer, a bidirectional Transformer encoder, a Linear layer, and a Softmax layer;

[0088] The housing price index prediction module is used to obtain the historical real estate price index, form a real estate index with the real estate sentiment index, and input the historical real estate index into the VAR model to obtain the real estate price index for the next period.

[0089] To verify the effectiveness of the real estate price index prediction provided by the present invention, an empirical study was carried out in four representative cities, Beijing, Shanghai, Guangzhou and Shenzhen.

[0090] Table 1 Prediction performance of the VAR model

[0091]

[0092] The prediction results are shown in Table 1. For the prediction of both new and second-hand housing price indices, the data of each index indicate the accuracy and reliability of the prediction. The average index data shows that the mean absolute error is 0.128, the root mean square error is 0.161, and the coefficient of determination is 0.884, showing good prediction performance as a whole.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting urban real estate price index based on online news sentiment analysis, characterized in that: include: Step A: collecting news text data according to keywords, preprocessing the news text data, and using a structured method to generate a standardized real estate online news data set; Step B: Calculate the news sentiment value corresponding to each news in the real estate online news dataset based on the BERT sentiment analysis framework, and calculate the real estate sentiment index based on the news sentiment value; including: The news text in the real estate news dataset is segmented, and the segmentation results are analyzed using the BERT pre-trained model to obtain the probability of each sentiment category to which the news text belongs. The sentiment type with the highest probability value is the news sentiment value, and the news sentiment values ​​are summarized by time to obtain the real estate sentiment index; the BERT pre-trained model includes a word vector embedding layer, a bidirectional Transformer encoder, a Linear layer, and a Softmax layer; Step C: Obtain a historical real estate price index, and form a real estate index with the real estate sentiment index, and input the historical real estate index into a VAR model to obtain a real estate price index for the next period.

2. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 1 is characterized in that: The preprocessing includes data denoising, format unification and invalid information elimination.

3. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 1 is characterized in that: The keywords include time range, city name, and real estate.

4. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 1 is characterized in that: The news text in the real estate news dataset is segmented, specifically including: The WordPiece algorithm is used to segment the news text X in the real estate news dataset into words W= <CLS,w1,w2,…,SEP,w i ,…,SEP>, where w i represents the i-th word in the news text, CLS represents the start marker of a sentence, and SEP represents the separator between sentences.

5. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 1 is characterized in that: The BERT pre-training model training process includes: Constructing a BERT pre-trained model, using a BERT model pre-trained with 250 million Chinese Wikipedia vocabulary data, the BERT model having 12 layers of bidirectional Transformer encoders, each layer equipped with 12 self-attention heads, and adding the Linear layer and the Softmax layer after the bidirectional Transformer encoder; Fine-tune the BERT pre-trained model, use the labeled news text dataset to train the BERT pre-trained model, and use the cross entropy loss as the loss function.

6. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 1, characterized in that: The step of aggregating the news sentiment values ​​by time to obtain the real estate sentiment index specifically includes: aggregating the news sentiment values ​​predicted by the BERT pre-training model by month, and calculating the real estate sentiment index by the following formula: Among them, d.label represents the news sentiment value corresponding to the news text, n represents the number of news texts in the news text dataset of the current month, r represents the real estate sentiment index, and timespan represents the time range of the current month.

7. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 1, characterized in that: The VAR model expression formula is as follows: in, represents the real estate index in time period t, c t represents the real estate price index in time period t, r t represents the real estate sentiment index in time period t, A0=[a c0 ,a h0 ]′ represents a constant vector, represents the parameter matrix, k represents the lag order and k≥i>1, ε t =[ε ct ,ε ht ]′ represents the residual vector.

8. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 7 is characterized in that: The ADF method is used to detect the stability of the predicted real estate sentiment index and real estate price index. If any index of the VAR model is non-stationary, the real estate index is subjected to differential processing. The processing formula is as follows: in, Represents the real estate index for time period t-1.

9. The method for predicting urban real estate price index based on online news sentiment analysis according to claim 7, characterized in that: The Akaike information criterion is used to solve the optimal lag order k of the VAR model, and the calculation formula is as follows: AIC(k)=-2*ln(L)+2*k Where L represents the likelihood value of the VAR model; Based on the determined lag order k, the least squares method is used to estimate the parameters A0, A i and ε t , the calculation formula is as follows: Where A=[A0,A1,…,A k ] represents the parameter matrix set, Y = [Y t-1 ,…,Y t-k ] represents the independent variable matrix; After obtaining A, substitute it into the VAR model to solve the residual vector ε t .

10. A device for predicting urban real estate price index based on online news sentiment analysis, characterized in that: include: A data collection module, used to collect news text data according to keywords, pre-process the news text data, and generate a standardized real estate online news data set using a structured method; The sentiment index prediction module is used to analyze the news sentiment value corresponding to each news in the real estate online news dataset based on the BERT sentiment analysis framework, and calculate the real estate sentiment index based on the news sentiment value; including: The news text in the real estate news dataset is segmented, and the segmentation results are analyzed using the BERT pre-trained model to obtain the probability of each sentiment category to which the news text belongs. The sentiment type with the highest probability value is the news sentiment value, and the news sentiment values ​​are summarized by time to obtain the real estate sentiment index; the BERT pre-trained model includes a word vector embedding layer, a bidirectional Transformer encoder, a Linear layer, and a Softmax layer; The housing price index prediction module is used to obtain the historical real estate price index and form a real estate index with the real estate sentiment index, and input the historical real estate index into the VAR model to obtain the real estate price index of the next period.

Images