Residential engineering cost prediction method based on LSTM-TRANSFORMER-RF

Abstract

The invention discloses a residential engineering cost prediction method based on LSTM-TRANSFORMER-RF, and relates to the field of engineering cost prediction. The method comprises the following steps: collecting residential single project cost data samples formed by multi-dimensional feature data at multiple time points in a historical time period to construct a training set; performing normalization preprocessing on the data in the training set; performing importance evaluation on each feature of the multi-dimensional features by using a random forest, respectively endowing each feature with a weight, generating new feature data based on the weight of each feature and original feature data in the training set, and adding the new feature data into the training set to obtain a new training set; establishing an LSTM-TRANSFORMER-RF mixed deep learning model, and training the LSTM-TRANSFORMER-RF mixed deep learning model by using the new training set to obtain a residential engineering cost prediction model; and the residential engineering cost prediction model is used for predicting the residential engineering cost in the residential construction process.

Description

Technical Field

[0001] The present invention relates to the field of project cost prediction, and in particular to a residential project cost prediction method based on LSTM-TRANSFORMER-RF. Background Art

[0002] The construction cost of a residential project refers to the total amount of expenses incurred during all stages of residential construction, from project planning, design, and construction to completion, acceptance, and commissioning. It encompasses costs such as building materials, labor, machinery and equipment, and administrative expenses. It is a key indicator for measuring the economic investment in residential construction, directly impacting developers' cost control and homebuyers' affordability. Accurately predicting construction costs is of great significance. For developers, it allows them to plan funds in advance, avoid overspending, and improve project profitability. For example, accurate estimates can reduce losses caused by construction delays due to insufficient funds. For homebuyers, it helps them understand housing price trends and make informed purchasing decisions. At the macro level, it facilitates government market regulation, ensures the healthy and stable development of the real estate market, and promotes the rational allocation of resources.

[0003] There are various methods for predicting residential construction costs, including empirical formulas, regression analysis, and artificial neural networks. Each of these methods has its own unique characteristics and is often combined to improve forecast accuracy. Residential construction cost forecasting faces numerous challenges. Forecasting methods often rely on historical data, making forecasts prone to inaccuracy if the market environment undergoes sudden changes, such as significant fluctuations in raw material prices. Model development often struggles to fully account for complex factors such as policy adjustments and natural disasters. Different methods are applicable to different scenarios, and choosing the wrong one can reduce forecast accuracy. Furthermore, data quality varies, with missing and erroneous data, further complicating forecasting. Consequently, residential construction cost forecasting is influenced by multiple factors and exhibits characteristics such as nonlinearity, volatility, and chaos. Traditional statistical models struggle to capture the nonlinearity and volatility of residential construction cost data, and single LSTM (Long Short-Term Memory) or TRANSFORMER models also have limitations. Existing technologies face two technical difficulties: First, existing feature construction methods inadequately represent nonlinear relationships and key features; and second, single model architectures suffer from modal limitations and information coupling flaws in cost forecasting. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a residential project cost prediction method based on LSTM-TRANSFORMER-RF.

[0005] The technical solution of the present invention is:

[0006] A residential project cost prediction method based on LSTM-TRANSFORMER-RF, the method includes the following steps:

[0007] Step 1: Collect residential single project cost data samples consisting of multi-dimensional feature data at multiple time points in the historical period, construct a residential single project cost data set, and divide the training set from it;

[0008] Step 2: Normalize and preprocess the data in the training set;

[0009] Step 3: Use random forest to evaluate the importance of each feature of the multidimensional feature, assign weights to each feature, generate new feature data based on the weights of each feature and the original feature data in the training set, and add the new feature data to the training set to obtain a new training set;

[0010] Step 4: Build an LSTM-TRANSFORMER-RF hybrid deep learning model: The LSTM branch and the TRANSFORMER branch are connected in parallel to collect feature vectors of the input data. The feature vectors extracted by the LSTM and Transformer branches are fused through a concatenation layer and then connected in series with the random forest model.

[0011] Step 5: Use the new training set to train the LSTM-TRANSFORMER-RF hybrid deep learning model established in step 4 to obtain a residential project cost prediction model;

[0012] Step 6: Use the residential project cost prediction model to predict the residential project cost during the residential construction process.

[0013] Furthermore, according to the residential project cost prediction method, the multidimensional features include delivery form, seismic resistance level, fortification intensity, structure type, foundation form, total construction area, floor area, above-ground construction area, underground construction area, number of above-ground floors, underground floor height, and standard above-ground floor height.

[0014] Furthermore, according to the residential project cost prediction method, the normalization preprocessing of the data in the training set described in step 2 is: normalizing all feature data and label data in the training set to the range of 0.1-1.1.

[0015] Furthermore, according to the residential project cost prediction method, the label data is a unit cost.

[0016] Furthermore, according to the residential project cost prediction method, the new feature data is generated based on the weight of each feature and the original feature data in the training set in step 3 as follows:

[0017]

[0018] in, represents the original feature data matrix in the training set, m represents the number of samples, n represents the number of features; ⊙ represents the Hadamard product; X i represents the i-th original feature data; w i Represents the weight value of the i-th original feature; then the feature data matrix in the new training set is

[0019] Compared with the existing technology, the present invention has the following beneficial effects:

[0020] (1) The present invention first uses a random forest model to evaluate the importance of the original features (12 dimensions) of the residential project cost, normalizes the importance weights, and then weights and reconstructs the original features of the residential project cost to generate aggregate features (13 dimensions) with physical meaning. This effectively alleviates the defect of the traditional linear weighted method in not capturing nonlinear relationships. The core advantages of this method are: enhancing the discrimination of important features of the residential project cost through data-driven weight distribution; the constructed aggregate features carry high-order combination information and form a cascade optimization with subsequent models, providing purer input features for the deep learning model.

[0021] (2) A trained LSTM-TRANSFORMER-RF hybrid model is proposed as a residential project cost prediction model. This hybrid model achieves the coordinated extraction of temporal features and global features through a dual-branch parallel structure. The LSTM (Longshort-term memory) branch processes the input sequence and extracts time-step features; the Transformer branch generates global features through a multi-head attention mechanism. The two are concatenated and then reduced in dimension through a fully connected layer. The core advantages of this method are: the short-term memory of the LSTM complements the long-term dependency of the Transformer; the feature fusion layer mines cross-modal associations through a nonlinear activation function.

[0022] (3) The residential project cost prediction model achieves the complementary advantages of "deep features + traditional models" through a two-stage integration strategy. In the first stage, LSTM-Transformer generates 128-dimensional fused features, and in the second stage, random forest (RF) is used for decision optimization. This method overcomes the limitations of a single model: deep features carry nonlinear spatiotemporal patterns, which reduces the discreteness of the RF feature importance distribution and improves decision reliability; RF's integrated learning characteristics effectively suppress the overfitting risk of deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the flow of the residential project cost prediction method based on LSTM-TRANSFORMER-RF in this embodiment;

[0024] Figure 2Schematic diagram of the structure of the LSTM-TRANSFORMER-RF hybrid model in this embodiment. DETAILED DESCRIPTION

[0025] To facilitate understanding of the present application, the present application will be described more comprehensively below with reference to the relevant drawings.

[0026] This paper provides a residential project cost prediction method based on LSTM-TRANSFORMER-RF. Its core concept is as follows: To address the inadequate representation of nonlinear relationships and key features in existing residential project cost prediction methods using feature construction methods, this paper proposes generating new features based on feature importance within the training set. Furthermore, to address the modal limitations and information coupling deficiencies of single model architectures in residential project cost prediction, this paper proposes a hybrid LSTM-TRANSFORMER-RF model with a dual-branch parallel structure that collaboratively extracts temporal and global features. Furthermore, this model utilizes a two-stage integration strategy to achieve the complementary advantages of "deep features + traditional models." In the first stage, LSTM-Transformer generates fused features for residential project cost, while in the second stage, RF is used for decision optimization.

[0027] Figure 1 This is a flow chart of the residential project cost prediction method based on LSTM-TRANSFORMER-RF in this embodiment. Figure 1 As shown, the method includes the following steps:

[0028] Step 1: Collect residential single project cost data samples composed of multidimensional feature data at multiple time points in the historical time period, construct a residential single project cost data set, and perform relevant preprocessing on the data in the data set to divide it into a training set and a validation set; the multidimensional features include delivery mode, seismic level, fortification intensity, structure type, foundation form, total construction area, land area, above-ground construction area, underground construction area, number of above-ground floors, underground floor height, and above-ground standard floor height.

[0029] The residential single project cost dataset can be a public residential single project cost dataset that can be obtained from the Internet, or it can be a residential single project cost dataset statistically collected from residential single project data collected by oneself. In this embodiment, residential single project data of less than 18 floors in Jiangsu Province in the past five years were collected from the Broad Group Index Network. A total of 165 data samples consisting of 12-dimensional feature data were statistically analyzed, which are represented as 165 rows and 12 columns of feature data to form a residential single project cost dataset. Then, the dataset was divided into 132 rows and 12 columns of feature data, and the training set was constructed with the unit cost as the label, that is, 165 rows and 1 column of label data. In this embodiment, non-data type features in the feature data are quantified, and each non-data type feature is converted into a numerical form that can be used for model calculation. The non-data type features are delivery form, seismic level, fortification intensity, structure type and foundation form. Table 1 shows an example of quantizing non-data type features into numerical values.

[0030] Table 1 Examples of non-data type features being quantified into specific values

[0031]

[0032] Step 2: Perform normalization preprocessing on both the feature data and label data in the training set.

[0033] The main function of data normalization is to eliminate the differences in the numerical values ​​and dimensions of different feature data and unify the data into the same scale range. This can avoid the adverse effects on model training caused by the excessive or insufficient values ​​of some feature data, improve the training effect and convergence speed of the model, and enhance the stability and generalization ability of the model. In this embodiment, if the feature data has a value of 0, if conventional normalization or standardization methods are used, such as the commonly used maximum-minimum normalization formula When x i =0 and x min = 0, the numerator is 0, which will cause the normalized value to be always 0, which will make the data lose its discrimination and make it difficult for the model to learn the effective information of the feature; in the normalization formula In, if x i =0, it will be affected by the mean μ and standard deviation σ, and the problem of data feature loss or distortion may also occur, seriously affecting the model training effect. To this end, this implementation adopts the strategy of normalizing all feature data and label data to the range of 0.1-1.1, and its calculation formula is:

[0034]

[0035] This approach not only effectively avoids the computational risks associated with zero values, but also preserves the original characteristics and differences of the data. Normalizing to the range of 0.1-1.1 prevents the data from being too concentrated near 0 after normalization, ensuring that every piece of data can play a role in model training and allowing data to be analyzed at the same scale, providing a more reliable data foundation for subsequent model training and prediction.

[0036] Step 3: Using the training set to train a random forest (RF) model, the importance of each feature of the multidimensional feature is evaluated and a new feature is generated based on the importance of each feature, and the new feature is added to the training set to obtain a new training set;

[0037] In data-driven prediction models, different features significantly influence the prediction results, varying in importance. In the context of residential construction cost prediction, features such as foundation type and structural type have a more critical impact on the unit cost, while less important features such as seismic rating, design intensity, and structural type have a relatively weaker influence. This difference in feature importance directly affects the model's prediction performance. Failure to properly distinguish and utilize these features can lead to inaccurate capture of key information in the data, resulting in biased predictions. Rationally strengthening high-importance features and de-emphasizing low-importance features is crucial and meaningful. This strategy allows the residential construction cost prediction model to focus more on key information, improve its sensitivity to important factors, and thus enhance its accuracy. It also helps reduce the interference of noisy data, optimize the model's performance, and prevent the model from falling into local optimal solutions due to excessive focus on minor features. This approach enables the residential construction cost prediction model to perform more stable and reliable predictions when faced with complex and changing data. The new feature generation formula is as follows:

[0038]

[0039] in, Represents the original feature data matrix, m represents the number of samples, and n represents the number of features; Represents the normalized feature weight ratio, feature_importance represents the importance of a single feature; ⊙ represents the Hadamard product (element-wise multiplication); X i represents the i-th original feature data in each feature data sample; w i represents the normalized weight value of the i-th original feature; X new Represents the new feature data matrix. The original feature data matrix and the new feature data matrix are combined to construct the final feature data matrix:

[0040] The specific implementation process is as follows:

[0041] (1) Input the original data in the training set into the random forest (RF) model to train the RF;

[0042] (2) Extract feature weights: Obtain feature importance and normalize:

[0043]

[0044] (3) Generate weighted features with a dimension of (132, 12):

[0045] X weighted =X⊙W

[0046] (4) Create an aggregate feature, add rows, and use a dimension of (132, 1), where n is 12:

[0047]

[0048] (5) Feature concatenation, dimension is (132,13):

[0049] X final =[XX new ]

[0050] Step 4: Build an LSTM-TRANSFORMER-RF hybrid deep learning model;

[0051] Figure 2 This is a schematic diagram of the structure of the LSTM-TRANSFORMER-RF hybrid model in this embodiment. Figure 2 As shown in FIG, the LSTM-TRANSFORMER-RF deep learning model proposed in this embodiment adopts a cascade architecture. The LSTM branch and the Transformer branch are connected in parallel to collect data features. Subsequently, the features are integrated through the splicing layer and then connected in series with the random forest (RF) model to finally output the results, as shown in FIG. Figure 2As shown in the figure. In this implementation, the LSTM branch consists of five stacked LSTM layers, each containing 64 hidden units. The LSTM-TRANSFORMER-RF deep learning model adopts a "dual-branch parallel + integrated optimization" architecture, with input consisting of three-dimensional time series data (sample size × sequence length × number of features, e.g., 165 × 1 × 12). First, the LSTM branch processes the input sequence, extracting local features at the last time step. Specifically, in this implementation, the output of the LSTM branch is the 64-dimensional feature vector extracted at the last time step. Simultaneously, the Transformer branch effectively maps the input 12-dimensional feature vector to a 64-dimensional space through linear projection. Positional encoding is then added, a crucial step that enables the model to accurately perceive and understand the positional information of each element in the sequence. The Transformer branch then further processes the features using a three-layer encoder, each with a four-head attention mechanism. This design helps the model comprehensively and deeply extract global features. Finally, average pooling is used to refine the encoder output sequence into a 64-dimensional representation vector. The features extracted by the LSTM and Transformer branches are then fused at the concatenation layer to form a 128-dimensional joint feature vector. This joint feature vector then undergoes three layers of mapping in a fully connected network: the first layer maps it to a 64-dimensional space, the second layer further maps it to 32 dimensions, and the final layer maps it to 1 dimension, outputting a preliminary prediction. Furthermore, the 128-dimensional fused features obtained from the concatenation layer are passed as input to a random forest model consisting of 150 decision trees. Leveraging its powerful optimization capabilities, the random forest model meticulously processes the 128-dimensional fused features, ultimately outputting a more accurate and reliable regression prediction.

[0052] Step 5: Use the new training set to train the LSTM-TRANSFORMER-RF hybrid deep learning model established in step 34 to obtain a residential project cost prediction model;

[0053] The training process is as follows:

[0054] (1) Input data processing; Input data size, each sample is regarded as a sequence of length 1, containing 12 features. If the batch size is 32, the input three-dimensional time series data is (32, 1, 12).

[0055] (2) After the input data (32, 1, 12) is processed by the LSTM branch, the output data is (32, 1, 64); the calculation formula is:

[0056] h t ,(c n ,h n )=LSTM(X)

[0057] in, The role of the LSTM branch is to capture temporal dependencies and extract local time step features.

[0058] (3) At the same time, the input data (32, 1, 12) is processed by the Transformer branch: first, the linear projection part effectively maps the input 12-dimensional feature vector to the 64-dimensional space (12→64), and the output data is (32, 1, 64). The calculation formula is:

[0059]

[0060] Then, position encoding is added, adding sine / cosine position information, and the output size is (32, 1, 64). Next, the Transformer branch further processes the features through a three-layer encoder. Each encoder layer contains a 4-head attention mechanism, and the output size is (32, 1, 64). The calculation formula is:

[0061] H trans =TransformerEncoder(Z+PositionalEncoding)

[0062] The role of the Transformer branch is to capture global feature interactions, eliminate position bias, and enhance long-range dependency modeling.

[0063] (4) Feature fusion and full connection prediction: The features (32, 64) extracted by the LSTM branch and the features (32, 64) extracted by the Transformer branch are concatenated and fused to form a 128-dimensional joint feature vector. This joint feature vector is reduced to 64 dimensions through the first fully connected layer (FC); then it is nonlinearly transformed through the ReLU activation function; then it is further compressed to 32 dimensions through the second fully connected layer and activated again through the ReLU; finally, the 32-dimensional features are mapped to 1-dimensional output through the third fully connected layer, and the output size is (32, 1). The calculation formula is:

[0064]

[0065] The role of the splicing layer is to fuse local time series and global features and generate preliminary prediction values ​​through nonlinear transformation.

[0066] (5) The 128-dimensional joint feature vector is used as the input of the random forest, the joint feature dimension ∈ (32, 128), and the label dimension ∈ (32, 1). First, feature extraction is performed to extract the joint feature features from the trained hybrid model. Then RF training is performed, and the training formula is:

[0067]

[0068] The role is to use joint features to enhance the generalization ability of RF and combine the advantages of deep features and traditional models.

[0069] Step 6: Use the residential project cost prediction model to predict the residential project cost during the residential construction process.

[0070] This implementation method verifies the residential project cost prediction model through a validation set, and adopts three widely used evaluation indicators: mean absolute error (MAE), root mean square error (RMSE) and coefficient of determination (R 2 ) to evaluate the prediction performance of the residential project cost prediction model. The mean absolute error (MAE) measures the average magnitude of the error between the predicted value and the actual value, regardless of direction. Its calculation formula is:

[0071]

[0072] where y i and Represent the true value and the predicted value respectively. MAE is robust to outliers and can be intuitively interpreted in raw units as the average error. The root mean square error (RMSE) calculates the square root of the mean squared difference between the predicted value and the observed value:

[0073]

[0074] Compared to MAE, RMSE penalizes larger errors more severely and is therefore sensitive to outliers. 2 ) quantifies the proportion of the variance of the dependent variable explained by the model:

[0075]

[0076] in is the mean of the observations. R 2 The value range is 0 to 1, with higher values ​​indicating better fit. However, if the model is overfitted, this indicator may be misleading. MAE and RMSE provide absolute measures of error (lower values ​​indicate better performance), while R 2 Provides a relative measure of goodness of fit. MAE is recommended when you need to avoid outliers dominating the evaluation, while RMSE is recommended when you emphasize large errors. 2 It is suitable for comparison of models of different magnitudes, but needs to be combined with error indicators for comprehensive evaluation.

[0077] The comparison algorithms used in this implementation include Random Forest (RF), SVR, a single LSTM, and a single Transformer. The following is a brief introduction to the four models. RF: A bagging method based on ensemble learning, consisting of multiple decision trees that output results through voting or averaging. It has strong resistance to overfitting and is suitable for high-dimensional data; it naturally supports parallel training and is computationally efficient; it can handle missing values ​​and provide feature importance assessment. Support Vector Regression (SVR) is a regression method based on support vector machines. It maps data into a high-dimensional space using a kernel function and searches for the optimal fit function that keeps the prediction error within the range of ε. It balances model complexity and generalization ability and is suitable for nonlinear and high-dimensional data. LSTM (Long Short-Term Memory) is a variant of RNN that addresses long sequence dependencies through a gating mechanism (input gate, forget gate, output gate). It excels at processing time series data (such as text, speech, and time series), can capture long-term dependencies, and avoids the vanishing / exploding gradient problem. Transformer is a model based on the self-attention mechanism, completely abandoning the RNN / CNN structure and processing sequence data in parallel. It has strong long-distance dependency modeling capabilities and high training efficiency (parallelization); it has excellent scalability. This embodiment prepares two training sets from the data set, a training set consisting of original feature data and a spliced ​​training set based on feature importance to generate new features. Different algorithms are used to test on these two data sets respectively. In order to comprehensively evaluate the performance of the model, this embodiment conducted multiple experiments. During the experiment, the parameters of the five regressors were optimized to determine their optimal parameter configuration. The optimal RF parameters are: n_estimators = 150, random_state = 42 to ensure the repeatability of the experiment. The default values ​​are selected for the parameters not specified. The parameters in LSTM and Transformer and the residential engineering cost prediction model remain consistent. The calculation results are shown in the table above. Among them, the new marked after the algorithm indicates the test conducted on the spliced ​​training set based on feature importance to generate new features. From the results, it can be seen that the advantages of the method for generating new features based on feature importance and the residential engineering cost prediction model of the present invention.

[0078] Table 2 Comparison of prediction performance of each model

[0079]

[0080]

[0081] It should be understood that, inspired by the technical concept of the present invention, those skilled in the art may make various improvements or changes based on the above content without departing from the content of the present invention, which still fall within the scope of protection of the present invention.

Claims

1. A residential construction cost prediction method based on LSTM-TRANSFORMER-RF, characterized by: The method comprises the following steps: Step 1: Collect residential single project cost data samples consisting of multi-dimensional feature data at multiple time points in the historical period, construct a residential single project cost data set, and divide the training set from it; Step 2: Normalize and preprocess the data in the training set; Step 3: Use random forest to evaluate the importance of each feature of the multidimensional feature, assign weights to each feature, generate new feature data based on the weights of each feature and the original feature data in the training set, and add the new feature data to the training set to obtain a new training set; Step 4: Establish an LSTM-TRANSFORMER-RF hybrid deep learning model; Step 5: Use the new training set to train the LSTM-TRANSFORMER-RF hybrid deep learning model established in step 4 to obtain a residential project cost prediction model; Step 6: Use the residential project cost prediction model to predict the residential project cost during the residential construction process.

2. The residential construction cost prediction method according to claim 1, characterized in that: The multidimensional characteristics include delivery form, seismic resistance level, design intensity, structure type, foundation form, total building area, floor area, above-ground building area, underground building area, number of above-ground floors, underground floor height, and standard above-ground floor height.

3. The residential construction cost prediction method according to claim 1, characterized in that: The normalization preprocessing of the data in the training set described in step 2 is: normalizing all feature data and label data in the training set to the range of 0.1-1.

1.

4. The residential construction cost prediction method according to claim 3, characterized in that: The label data is a unilateral cost.

5. The residential construction cost prediction method according to claim 3, characterized in that: The new feature data generated based on the weight of each feature and the original feature data in the training set in step 3 is as follows: in, represents the original feature data matrix in the training set, m represents the number of samples, n represents the number of features; ⊙ represents the Hadamard product; X i represents the i-th original feature data; w i Represents the weight value of the i-th original feature; then the feature data matrix in the new training set is 6. The residential construction cost prediction method according to claim 1, characterized in that: The LSTM-TRANSFORMER-RF hybrid deep learning model described in step 4 is as follows: the LSTM branch and the TRANSFORMER branch are connected in parallel to collect feature vectors of the input data. The feature vectors extracted by the LSTM and Transformer branches are fused through the splicing layer and then connected in series with the random forest model.

Images