A large model-based automatic extraction and analysis method for engineering cost

Abstract

The application provides a large model-based automatic extraction and analysis method for engineering cost, and belongs to the technical field of engineering cost data analysis. First, a text of an engineering project to be analyzed is obtained, the text of the engineering project is input into a constructed and trained engineering cost analysis large model, and a specific cost problem is input; the engineering cost analysis large model processes and analyzes the text of the engineering project to obtain various cost information of the corresponding project, and then generates a corresponding reply to the specific input cost problem to realize automatic extraction and analysis of the engineering cost; the engineering cost analysis large model comprises a word embedding layer, a multilayer encoder module, a multilayer decoder module and an output layer; each corresponding layer of the multilayer encoder module and the multilayer decoder module is connected through a skip connection to form a U-Net architecture; the application can automatically adapt to different data formats and complex scenarios, and does not need to manually design rules and templates, and can accurately analyze specific cost problems.

Description

Technical Field

[0001] This invention belongs to the field of engineering cost data analysis technology, and in particular relates to an automatic extraction and analysis method for engineering costs based on a large model. Background Technology

[0002] As infrastructure construction and engineering projects expand in scale, their complexity and diversity increase. These projects involve significant resource allocation, cost control, and schedule management, placing higher demands on cost management. Therefore, accurate cost management and control are crucial for project success in current construction processes. However, traditional cost management methods rely heavily on human experience and manual operations, which struggle to handle the complex and ever-changing data in modern engineering projects. This leads to inaccurate cost estimations, untimely information updates, and difficulties in making scientific decisions. These problems not only increase the risk of cost overruns and schedule delays but also negatively impact overall project benefits. Therefore, the development of automated cost extraction and analysis technologies can significantly improve work efficiency and data accuracy, reduce errors in manual operations, and thus lower the risk of cost overruns in engineering projects. Furthermore, the application of this technology can provide modern engineering projects with more efficient, accurate, and scientific cost management tools, enhancing overall project management and economic benefits.

[0003] Existing methods for automatic extraction and analysis of engineering costs typically rely on rule-based cost analysis systems, machine learning-based analysis methods, and deep learning-based cost analysis methods. While these methods can automate the processing and analysis of engineering cost data to some extent, they still have some key drawbacks.

[0004] Rule-based cost analysis systems extract project cost information through predefined rules and templates, relying on the knowledge of domain experts for design and maintenance. However, this approach exhibits significant limitations when dealing with different data formats and complex scenarios, lacking flexibility and versatility, while also incurring high maintenance costs and difficulty in handling complex or unstructured data.

[0005] Machine learning-based analytical methods: Traditional machine learning algorithms, such as decision trees and support vector machines, are trained using hand-designed features to predict and classify engineering cost data. While these methods perform well on simple tasks, their feature extraction process relies on the designer's experience, making it difficult to capture deep relationships in complex data. Furthermore, they exhibit weak scalability and generalization ability when faced with large-scale, diverse data.

[0006] Deep learning-based cost analysis methods: This method utilizes complex neural network models to automatically extract features from engineering project data, improving the accuracy and efficiency of cost analysis to some extent. However, it may not effectively adapt or transfer to different types of engineering data or cross-domain applications, limiting its applicability in complex real-world application environments. It may require adjustments or retraining for each specific task to meet specific needs. Summary of the Invention

[0007] To address the above problems, this invention provides a method for automatic extraction and analysis of engineering costs based on a large model, comprising the following steps:

[0008] Obtain the project text to be analyzed, which contains key information related to cost analysis;

[0009] The engineering project text is input into the constructed and trained large-scale engineering cost analysis model, along with specific cost questions. The large-scale engineering cost analysis model processes and analyzes the engineering project text to obtain various cost information for the corresponding project. Then, it generates corresponding answers to the specific input cost questions, thereby realizing the automatic extraction and analysis of engineering costs.

[0010] The large-scale engineering cost analysis model includes a word embedding layer, a multi-layer encoder module based on a multi-head attention mechanism, and a multi-layer decoder module, as well as an output layer; each corresponding layer of the multi-layer encoder module and the multi-layer decoder module forms a U-Net architecture through skip connections;

[0011] The word embedding layer is used to obtain word encoding direction and position encoding vectors and generate word embedding results; the multi-layer encoder module is used to deeply extract hierarchical features of the data. Each encoder layer takes the output of the previous layer as input, enabling the model to capture more complex features and patterns; the multi-layer decoder module incorporates the features of the corresponding encoder layer into the input of each decoder layer through skip connections and multi-head attention, decoding and restoring complex features layer by layer. Each decoder layer further fuses the features of the encoder for feature extraction and optimization; the output layer is used to transform the representation generated by the decoder back into the probability distribution in the vocabulary.

[0012] Preferably, the process of collecting the original dataset for pre-training the large-scale engineering cost analysis model includes: filtering out key information related to cost analysis in project documents through keyword matching, including quantitative clauses in the contract, budget details in the bidding documents, material specifications, material usage, and labor costs; at the same time, in order to fully improve the automatic cost extraction and analysis capabilities by utilizing the large-scale model's ability to process contextual information, the project scale, geographical location, construction conditions, and other contextual information are also included as original text data.

[0013] Preferably, the specific process of labeling the original dataset is as follows:

[0014] The system provides corresponding data annotations for different cost issues, including detailed information related to engineering costs. Based on the needs of engineering cost analysis, the data is annotated at multiple levels, with specific cost items annotated at a fine-grained level and the overall cost structure of the project annotated at a coarse-grained level. Each set of issues is labeled with the corresponding region, so that during the pre-training of the large model, the system can automatically adapt to the local units, cost structure, and cost categories for engineering projects in different regions.

[0015] Preferably, the word embedding layer includes word encoding and position encoding, and the specific processing procedure is as follows:

[0016] First, the text is decomposed and encoded using the WordPiece word embedding method, and the word encoding vectors corresponding to the text data are obtained:

[0017]

[0018] in, The text data to be encoded. This is the encoded word vector. For word embedding functions;

[0019] Secondly, the position code is calculated using sine and cosine functions, namely:

[0020]

[0021] in, This indicates the position of each word in the text sequence. For encoding dimension index, The dimension for location encoding. This represents the result of the position encoding;

[0022] Finally, word encoding vectors with the same data dimensions will be used. and position encoding vector Element-wise addition is performed to generate word embedding results. .

[0023] Preferably, the multi-layer encoder module is based on a multi-head attention mechanism, a fully connected layer, a Dropout layer, and a layer standardization design for each encoder structure. The specific processing procedure is as follows:

[0024] Word embedding vectors based on multi-head attention mechanism Feature processing is performed to enable the encoder to capture key features in the engineering cost data from different perspectives; calculation , and matrix:

[0025]

[0026] in, , and These are query, key, and value functions, respectively. , and They represent the first time. The query, key, and value matrix under each attention head, and the total number of multi-head attention heads is... ,but ;

[0027] Next, multi-head attention features are calculated. :

[0028]

[0029] in, For the softmax function, Normalization factor;

[0030] Based on features By concatenating and processing the feature vectors of each head, information from multiple perspectives is integrated, enhancing the diversity and expressive power of features. The results of multi-head attention processing... The calculation is as follows:

[0031]

[0032]

[0033] in, No. The feature processing vectors of each attention head, and ; For channel splicing operations;

[0034] Layer standardization is performed to reduce the gradient vanishing problem when the model processes complex cost data, i.e.:

[0035]

[0036] in, These are the standard features after layer standardization. This is a layer normalization processing function;

[0037] Then, through fully connected layers, Dropout layers, and residual connections, further extracted features are obtained. :

[0038]

[0039]

[0040] in, and These represent the processing functions for the fully connected layer and the Dropout layer, respectively. This represents the extracted encoder intermediate features;

[0041] Finally, the feature processing results of the first-layer encoder are obtained again through layer normalization. :

[0042]

[0043] The constructed encoder structure is stacked to form a 10-layer structure, where the input of each layer encoder is the output of the layer above it; the output of the last layer encoder... This represents the overall processing result of the multi-layer encoder module.

[0044] Preferably, the multi-layer decoder module is based on a multi-head attention mechanism, a fully connected layer, a Dropout layer, and layer normalization design for each encoder structure. The specific processing procedure is as follows:

[0045] The output of the multi-layer encoder module As the vector to be decoded, and the encoded vector being the result of the encoder layer corresponding to this decoder layer, the encoder features are directly introduced into the decoder through skip connections to more accurately decode the engineering cost data. The processing results after multi-head attention are as follows:

[0046]

[0047]

[0048] in, It is an encoding vector, and ; and These are the results of multi-head attention processing for the encoded vector and the vector to be decoded, respectively.

[0049] Will and Perform addition and then process using layer normalization:

[0050]

[0051] in, These are the features after normalization processing at the decoder layer;

[0052] Further extracted features are obtained through fully connected layers, Dropout layers, and residual connections. :

[0053]

[0054]

[0055] in, This represents the extracted intermediate features from the decoder;

[0056] The feature processing result of the first layer decoder is obtained again through layer normalization. ,Right now:

[0057]

[0058] The constructed decoder structure is stacked, having the same number of layers as the multi-layer encoder structure. The input vector to each subsequent decoder layer is the output of the previous layer, and the input encoded vector to each decoder layer is the output of the corresponding encoder layer. The output of the last decoder layer... This represents the overall processing result of the multi-layer decoder module.

[0059] Preferably, after the multi-layer decoder obtains the final processing result, the output layer transforms the representation generated by the decoder back into the probability distribution in the vocabulary. The output layer consists of a fully connected layer and a softmax function, i.e.:

[0060]

[0061] in, The final word vector output is... This represents the softmax processing function.

[0062] Preferably, for the completed large-scale engineering cost analysis model, pre-training is first performed to obtain a preliminary professional scenario large-scale model adapted to engineering cost scenarios. Then, fine-tuning training is performed based on engineering cost problem data to obtain a professional large-scale model for automatic extraction and analysis of engineering costs. The fine-tuning training adopts an asymptotic fine-tuning strategy, gradually adapting to the data characteristics unique to engineering costs by unfreezing the network parameters layer by layer, while retaining the general knowledge learned in the pre-training stage.

[0063] Preferably, based on the collected project engineering text ,and Training is performed using an autoregressive language modeling method, that is, through each segment Given a paragraph of text, predict the next word; and define a pre-trained autoregressive loss function. for:

[0064]

[0065] in, For the first The number of autoregressive predictions corresponding to the text segment. Indicates the current word, Indicates in Previous words, These are the model parameters, which are obtained through random initialization.

[0066] The loss value is calculated based on randomly initialized model parameters, and the Adam optimization algorithm is used to optimize the model parameters and update the gradients.

[0067]

[0068] in, This represents the updated model parameters. Set the learning rate; and set an adaptive learning rate for model training: set a lower learning rate for shallow layers and a higher learning rate for deeper layers to ensure the stability of the model training process;

[0069] Then, the parameters are iteratively updated until the preset maximum number of iterations is reached. Then stop pre-training and save the optimized trained model as the pre-trained model.

[0070] Preferably, the issue of project cost should be addressed. As input to the model, and to generate corresponding information data. To train the model for the target, fine-tune the text generation loss function accordingly. Designed as follows:

[0071]

[0072] in, and Representing text respectively and The corresponding word vectors, This represents the model parameters during fine-tuning training, which are initialized to the pre-trained model parameters.

[0073] The training of shallow layer parameters is unfrozen layer by layer using an asymptotic fine-tuning strategy. The specific steps include:

[0074] Model parameter initialization :

[0075]

[0076] in, These represent the first to the second layers of the model, respectively. The network layer parameters are initialized to the pre-trained model parameters, and all weight parameters are frozen.

[0077] Unfreeze the last layer of the model and perform fine-tuning training and parameter updates to adapt it to the specific engineering cost analysis task. Then, further optimize and update the model parameters.

[0078]

[0079] in, Indicates the first The updated model parameters after the layer update The learning rate is consistent with the pre-training process; it is updated cyclically. The parameter update for the final layer is completed after the next iteration;

[0080] Continue by unfreezing the penultimate layer of the model and continuing fine-tuning training and parameter updates, namely:

[0081]

[0082] in, and The first The parameters of the layer network before and after the update, at this time Continue to update in a loop. The parameter update will be completed after the next iteration;

[0083] Continue thawing The training strategy involves progressively fine-tuning the network parameters layer by layer, starting from the first layer of the model.

[0084] Once all layer parameters are unfrozen, the model enters the global fine-tuning phase. At this point, all parameters are optimized simultaneously to reach the preset maximum number of fine-tuning training iterations. Then stop fine-tuning the model training.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] (1) Efficient and flexible automatic analysis of engineering cost: This invention utilizes large model technology and WordPiece word embedding and multi-head attention mechanism to effectively process and analyze complex engineering cost data. Compared with traditional analysis methods, this invention can automatically adapt to different data formats and complex scenarios, and does not require manual design of rules and templates, which significantly improves the flexibility and adaptability of the system.

[0087] (2) Large-scale engineering cost data feature extraction: Based on a large-scale engineering project database, this invention can deeply extract and analyze the actual cost information of a wide range of engineering projects; at the same time, based on the multi-level feature extraction method of multi-layer encoder and multi-layer decoder and skip connection, it can better capture the deep relationship between complex data, thereby making full use of existing engineering project data to provide reliable engineering cost analysis.

[0088] (3) More professional cost analysis model: This invention uses large-scale pre-training and model fine-tuning training based on progressive fine-tuning training strategy to enable the model to not only adapt to specific engineering scenarios, but also to perform accurate analysis when facing specific cost problems; and the model’s real-time feedback capability ensures rapid response and accurate analysis of new project data, improving the real-time performance and reliability of engineering cost analysis.

[0089] (4) Improve the intelligence level of cost analysis: Compared with existing methods, the model of the present invention can automatically extract and analyze cost information through online language interaction, reduce the need for manual intervention and improve the level of automation, thereby making the extraction and analysis of engineering costs more intelligent and efficient.

[0090] (5) Efficient Information Fusion and Transmission: The large-scale model designed based on the U-Net architecture in this invention utilizes the encoder-decoder structure and its skip connections to efficiently capture and fuse information at different levels in engineering cost texts, from words to sentences to paragraphs, thereby gaining a more comprehensive understanding of the content of engineering cost texts. Specifically, the skip connections in U-Net allow low-level information to be directly transmitted to higher levels, helping to recover more detailed text features during decoding, improving the accuracy of text extraction and analysis, and enhancing the accuracy of engineering cost analysis. Through the stacking of multiple encoders and decoders, the model can progressively extract and integrate deep-level features in the text. This hierarchical structure helps the model better understand the complex relationships and contextual information in engineering cost texts. Compared with large-scale models based on the Transformer architecture, the large-scale model designed based on the U-Net architecture in this invention adopts a more direct information transmission method during encoding and decoding, reducing unnecessary computational overhead and requiring relatively less training data. In the field of engineering cost, obtaining high-quality labeled data is difficult; therefore, compared with large-scale models based on the Transformer architecture, the large-scale model proposed in this invention is more suitable for automatic extraction and analysis tasks in engineering cost. Attached Figure Description

[0091] Figure 1 This is a block diagram of the overall structure of the large-scale engineering cost analysis model of this invention.

[0092] Figure 2 This is a block diagram of the encoder model structure of the multilayer encoder module of the present invention.

[0093] Figure 3 This is a block diagram of the decoder model structure of the multi-layer decoder module of the present invention.

[0094] Figure 4 This is a flowchart illustrating the overall process of large model pre-training and fine-tuning training in this invention. Detailed Implementation

[0095] The present invention will be further described below with reference to embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0096] The overall process of this invention is as follows:

[0097] Construction of Engineering Cost Database: Based on historical engineering project documents and various material costs and cost data, engineering cost data is constructed, and the raw data is cleaned and formatted to obtain an engineering cost database containing real project data.

[0098] Design a large-scale engineering cost analysis model: Based on the WordPiece word embedding method and multi-head attention mechanism, a large-scale engineering cost analysis model is constructed, which consists of a word embedding layer, a multi-layer encoder module, a multi-layer decoder module, and an output layer.

[0099] Model pre-training and fine-tuning training based on a large engineering cost analysis model: The engineering cost database is used to pre-train the engineering cost model to obtain a large language model that adapts to the engineering scenario. Then, fine-tuning training is performed on the engineering cost problem to obtain a professional large model for automatic extraction and analysis of engineering costs.

[0100] Model Deployment and Optimization: The trained large-scale model for automatic extraction and analysis of engineering costs is deployed to the backend server. Online language interaction is used to enable the large-scale model to automatically extract and analyze the costs of engineering projects, and the analysis results of the large-scale model are fed back to the user in real time. At the same time, the large-scale model is improved and fine-tuned using new engineering project data, thereby continuously improving the real-time accuracy of the model and the reliability of engineering cost analysis, ultimately meeting the needs of actual application scenarios.

[0101] I. Construction of the Engineering Cost Training Dataset

[0102] 1. Raw Text Data Collection: In the raw data collection phase, key information related to cost analysis in project documents is filtered out through keyword matching. This includes comprehensive textual information such as quantitative clauses in contracts, budget details in tender documents, material specifications, material usage, and labor costs. Simultaneously, to leverage the large model's ability to process contextually related information and fully enhance automated cost extraction and analysis capabilities, contextual information such as project scale, geographical location, and construction conditions is also included as raw text data. This ensures that the collected textual data not only covers cost details but also the background factors influencing these details. This related information will serve as supplementary data, providing a more comprehensive perspective for subsequent analysis.

[0103] 2. Data Cleaning and Preprocessing: Non-cost-related project descriptions, lengthy legal clauses, and technical parameters irrelevant to cost are filtered and removed from the original text data to reduce invalid data and improve dataset quality. Furthermore, imperial units are converted to metric units, currency symbols are standardized, and potential time format differences are handled to ensure data consistency. The cleaned text data is then uniformly converted to UTF-8 encoded plain text format and segmented to obtain a total of... The text data related to the project engineering, namely:

[0104] (1)

[0105] in, For complete text data, For the first Section project engineering text, and ;

[0106] 3. Data annotation: In order to achieve automated engineering cost extraction and analysis, extensive data annotation is required for cost data to meet engineering cost requirements;

[0107] Specifically, for different cost-related questions, professionals are organized to annotate the data accordingly. These questions include detailed information related to project costs, such as: What is the budget for building materials during construction? What is the budget for labor costs? What are the equipment rental and maintenance costs? When is the expected completion date of the project? Furthermore, based on the needs of project cost analysis, the data is annotated at multiple levels. For example, at a fine-grained level, specific cost items, such as "labor costs" and "material costs," are annotated; at a coarse-grained level, the entire project cost structure, such as "direct costs" and "indirect costs," is annotated. This multi-level annotation strategy helps the model use the annotated data more flexibly in different analysis tasks. In addition, each set of questions is annotated with the corresponding region, so that during model training, the large model can automatically adapt to local units, cost structures, and cost categories for projects in different regions. Corresponding project information This constitutes a set of engineering cost data ;

[0108] Data augmentation: Data augmentation involves expanding the collected engineering cost data through methods such as rephrasing, replacing synonyms, and adjusting text structure. For example, replacing "material cost" with "material expense" or "material expenditure" yields diverse data with the same semantic meaning but different expressions. This method effectively improves the model's ability to understand different expressions, ensuring its generalization ability and ultimately leading to... Group project cost data:

[0109] (2)

[0110] in, Indicates the first Group project cost data, and .

[0111] II. Construction of a Large-Scale Model for Engineering Cost Analysis

[0112] The overall structure of the large-scale engineering cost analysis model is as follows: Figure 1 As shown, the architecture includes a word embedding layer, a multi-layer encoder module based on a multi-head attention mechanism, and a decoder module, as well as an output layer. Each corresponding layer of the multi-layer encoder and decoder is connected via skip connections to form a U-Net architecture, directly concatenating the output feature map of the encoder to the input feature map of the corresponding decoder. Skip connections can alleviate the gradient vanishing problem, promote the effective propagation of gradients in deep networks, and thus accelerate model convergence. At the same time, they can also help preserve the original input information, thereby preserving the details and local information of the encoder features during the decoding process, ultimately improving the accuracy of the model in extracting and analyzing cost information.

[0113] 1. Word embedding layer

[0114] The word embedding layer includes word encoding and positional encoding. First, this invention decomposes and encodes the text based on the WordPiece word embedding method, and obtains the word encoding vector corresponding to the text data:

[0115] (3)

[0116] in, The text data to be encoded. This is the encoded word vector. For word embedding functions;

[0117] Secondly, the position code is calculated using sine and cosine functions, namely:

[0118] (4)

[0119] in, This indicates the position of each word in the text sequence. For encoding dimension index, The dimension for location encoding. This represents the result of the position encoding;

[0120] Finally, word encoding vectors with the same data dimensions will be used. and position encoding vector Element-wise addition is performed to generate word embedding results. ,Right now:

[0121] (5).

[0122] 2. Multilayer encoder module

[0123] The multi-layer encoder module is based on a multi-head attention mechanism, fully connected layers, dropout layers, and layer normalization design for each encoder structure. This multi-layer encoder module can deeply extract hierarchical features from the data. Each encoder layer uses the output of the previous layer as input, enabling the model to capture more complex features and patterns, thereby improving the accuracy and reliability of engineering cost analysis. The model structure diagram for each encoder layer is shown below. Figure 2 As shown.

[0124] The specific data processing procedure of the multilayer encoder module is as follows:

[0125] (1) Word embedding vectors based on multi-head attention mechanism Feature processing enables the encoder to capture key features in engineering cost data from different perspectives, improving its ability to understand complex information; computation... , and matrix:

[0126] (6)

[0127] in, , and These are query, key, and value functions, respectively. , and They represent the first time. The query, key, and value matrix under each attention head, and the total number of multi-head attention heads is... ,but .

[0128] (2) Then, calculate the multi-head attention features. :

[0129] (7)

[0130] in, For the softmax function, Normalization factor;

[0131] (3) Feature-based By concatenating and processing the feature vectors of each head, the encoder can integrate information from multiple perspectives, enhancing the diversity and expressive power of features, thus improving the multi-head attention processing results. The calculation is as follows:

[0132] (8)

[0133] (9)

[0134] in, No. The feature processing vectors of each attention head, and ; For channel splicing operations, the multi-head attention processing function is:

[0135] (10)

[0136] Among them, the multi-head attention processing function Specifically, this describes the processing steps from (1) to (3);

[0137] (4) Perform layer standardization to reduce the gradient vanishing problem when the model is dealing with complex cost data, i.e.:

[0138] (11)

[0139] in, These are the standard features after layer standardization. This is a layer normalization processing function;

[0140] (5) Then, through a fully connected layer, a Dropout layer, and a residual connection, further extracted features are obtained. :

[0141] (12)

[0142] (13)

[0143] in, and These represent the processing functions for the fully connected layer and the Dropout layer, respectively. This represents the extracted encoder intermediate features;

[0144] (6) Finally, the feature processing results of the first layer encoder are obtained again through layer normalization. :

[0145] (14)

[0146] Therefore, the overall process of feature processing by the encoder is as follows:

[0147] (15)

[0148] in, This represents the encoder layer processing function, which specifically refers to the processing steps from (1) to (6);

[0149] (7) Multilayer encoder structure: Based on the constructed encoder structure, stacking is performed. In this embodiment, there are a total of 10 layers, and the input of each layer encoder is the output of the layer above it; therefore, the output of each layer encoder is respectively , ,..., And the output of the last encoder layer This represents the overall processing result of the multi-layer encoder module.

[0150] 3. Multi-layer decoder module

[0151] Similarly, each encoder structure is designed based on multi-head attention, fully connected layers, dropout layers, and layer normalization. However, in the input of each decoder layer, features from the corresponding encoder layer are additionally incorporated through skip connections and multi-head attention, decoding and restoring complex features layer by layer. Each decoder layer further fuses the encoder features and extracts and optimizes features through fully connected layers, dropout layers, and layer normalization. This structure ensures full utilization of information during the decoding process, improving the model's ability to understand and analyze engineering cost data. The model structure diagram for each decoder layer is shown below. Figure 3 As shown.

[0152] The specific data processing procedure of the multi-layer decoder module is as follows:

[0153] (1) In this embodiment, the output of the multilayer encoder module is... The vector to be decoded is the encoder layer processing result corresponding to the decoder layer. That is, the first decoder layer corresponds to the processing result of the tenth encoder layer (the second decoder layer corresponds to the processing result of the ninth encoder layer, and so on). By directly introducing the encoder features into the decoder through skip connections, the multi-layer decoder can more accurately decode the engineering cost data and ensure the continuity of information transmission. The processing results after multi-head attention are as follows:

[0154] (16)

[0155] (17)

[0156] in, It is an encoding vector, and ; and These are the results of multi-head attention processing for the encoded vector and the vector to be decoded, respectively.

[0157] (2) and Perform addition and then process using layer normalization:

[0158] (18)

[0159] in, These are the features after normalization processing at the decoder layer;

[0160] (3) Further extracted features are obtained through fully connected layers, Dropout layers, and residual connections. :

[0161] (19)

[0162] (20)

[0163] in, This represents the extracted intermediate features from the decoder;

[0164] (4) The feature processing results of the first layer decoder are obtained again through layer normalization. This helps the model output more standardized features, ensuring that the decoder maintains stability and accuracy during feature transmission at different levels.

[0165] (twenty one)

[0166] Therefore, the overall process of feature processing by the encoder is as follows:

[0167] (twenty two)

[0168] in, This represents the decoder layer processing function, which specifically refers to the processing steps from (1) to (4);

[0169] (5) Multi-layer decoder structure: Based on the constructed decoder structure, it is stacked, with the same number of layers as the multi-layer encoder structure, i.e., 10 layers. The input vector to be decoded for each subsequent decoder layer is the output of the previous decoder layer, and the input encoded vector for each decoder layer is the output of the corresponding encoder layer. Therefore, the output of each decoder layer is as follows: , ,..., The output of the last decoder layer This represents the overall processing result of the multi-layer decoder module.

[0170] 4. Output Layer

[0171] After the multi-layer decoder obtains the final processing result, the representation generated by the decoder needs to be transformed back into the probability distribution in the vocabulary. The output layer consists of a fully connected layer and a softmax function, i.e.:

[0172] (twenty three)

[0173] in, The final word vector output is... This represents the softmax processing function.

[0174] III. Training of a Large-Scale Engineering Cost Analysis Model

[0175] Based on the constructed large-scale engineering cost analysis model, the first step is to pre-train the model to obtain a preliminary professional scenario-based large-scale model adapted to engineering cost scenarios. Then, fine-tuning is performed based on engineering cost problem data to obtain a specialized large-scale model for automatic extraction and analysis of engineering costs. The fine-tuning training employs an asymptotic fine-tuning strategy, gradually adapting the network parameters to the unique characteristics of engineering cost data by progressively unfreezing them layer by layer, while retaining the general knowledge learned in the pre-training stage. This improves the model's accuracy in analyzing different engineering cost problems. Furthermore, this strategy optimizes computational resource consumption during training, making the model's application in engineering cost analysis more efficient and accurate. The overall process is as follows: Figure 4 As shown.

[0176] 1. Large model pre-training

[0177] Collected project engineering text data Used for pre-training of large-scale models for engineering cost analysis; specifically, based on collected project engineering texts. ,and Training is performed using an autoregressive language modeling method, that is, through each segment Given a paragraph of text, predict the next word; and define a pre-trained autoregressive loss function. for:

[0178] (twenty four)

[0179] in, For the first The number of autoregressive predictions corresponding to the text segment. Indicates the current word, Indicates in Previous words, These are the model parameters, which are obtained through random initialization.

[0180] The loss value is calculated based on randomly initialized model parameters, and the Adam optimization algorithm is used to optimize the model parameters and update the gradients.

[0181] (25)

[0182] in, This represents the updated model parameters. Set the learning rate; and set an adaptive learning rate for model training: set a lower learning rate for shallow layers and a higher learning rate for deeper layers to ensure the stability of the model training process;

[0183] Then, the parameters are iteratively updated until the preset maximum number of iterations is reached. Then stop pre-training and save the optimized trained model as the pre-trained model.

[0184] 2. Model fine-tuning training

[0185] Collected engineering cost data Used for fine-tuning training of large-scale engineering cost analysis models, among which Indicates the first Group project cost data, and , Specifically, the issue of project cost. As input to the model, and to generate corresponding information data. The model is trained to address various engineering cost issues, ultimately enabling it to automatically generate corresponding cost information. Therefore, the text generation loss function is fine-tuned accordingly. Designed as follows:

[0186] (26)

[0187] in, and Representing text respectively and The corresponding word vectors, This represents the model parameters during fine-tuning training, which are initialized to the pre-trained model parameters.

[0188] The model fine-tuning training employs an asymptotic fine-tuning strategy, gradually unfreezing the training of shallow layer parameters. Specific steps include:

[0189] (1) Model parameter initialization :

[0190] (27)

[0191] in, These represent the first to the second layers of the model, respectively. The network layer parameters are initialized to the pre-trained model parameters, and all weight parameters are frozen (remain unchanged).

[0192] (2) The last layer of the model (the first layer) The (layer) network is unfrozen and fine-tuned through training and parameter updates to initially adapt it to a specific engineering cost analysis task, followed by further parameter optimization and updates to the model parameters:

[0193] (28)

[0194] (3) Among them, Indicates the first The updated model parameters after the layer update The learning rate is consistent with the pre-training process; it is updated cyclically. The parameter update for the final layer is completed after the next iteration;

[0195] (4) Continue with the second to last layer of the model (the... The (layer) network is unfrozen and fine-tuning training and parameter updates continue, i.e.:

[0196] (29)

[0197] in, and The first The parameters of the layer network before and after the update, at this time Continue to update in a loop. The parameter update will be completed after the next iteration;

[0198] (5) Continue thawing the first The training strategy involves progressively fine-tuning the model's network parameters layer by layer, starting from the first layer. By gradually unfreezing and optimizing the model parameters, the model can more accurately adapt to the characteristics of engineering cost data while retaining pre-training knowledge, effectively avoiding overfitting and ultimately improving the model's analytical accuracy on various engineering cost issues. Furthermore, this strategy can optimize resource utilization efficiency during training and reduce training time and computational resource consumption.

[0199] (6) After all the parameters of all layers are unfrozen, the model enters the global fine-tuning stage. At this time, all parameters are optimized uniformly to reach the preset maximum number of fine-tuning training iterations. Then stop fine-tuning the model; save the fine-tuned model as the final large model for engineering cost analysis, and use it as the final deployment model for engineering cost analysis.

[0200] IV. Model Deployment and Application

[0201] Large-scale model deployment: Select suitable server hardware configuration and deploy the trained engineering cost analysis large-scale model to the backend server cluster;

[0202] Input text: Users input the text of the engineering project to be analyzed into the large-scale engineering cost analysis model for extraction and analysis, and input specific cost issues;

[0203] Large model processing and analysis: First, the engineering project text is processed and analyzed to obtain the corresponding cost information of the project. Then, corresponding answers are generated for the specific input cost questions to realize the automatic extraction and analysis of engineering costs.

[0204] Model optimization: The large model is improved and fine-tuned in real time based on new project data, so that the large model can be updated and optimized synchronously, ultimately ensuring the real-time accuracy and reliability of engineering cost analysis to meet the needs of actual application scenarios.

[0205] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0206] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A large model-based automatic extraction and analysis method for engineering cost, characterized in that, Includes the following processes: Obtain the project text to be analyzed, which contains key information related to cost analysis; The engineering project text is input into the constructed and trained large-scale engineering cost analysis model, along with specific cost questions. The large-scale engineering cost analysis model processes and analyzes the engineering project text to obtain various cost information for the corresponding project. Then, it generates corresponding answers to the specific input cost questions, thereby realizing the automatic extraction and analysis of engineering costs. The large-scale engineering cost analysis model includes a word embedding layer, a multi-layer encoder module based on a multi-head attention mechanism, and a multi-layer decoder module, as well as an output layer; each corresponding layer of the multi-layer encoder module and the multi-layer decoder module forms a U-Net architecture through skip connections; The multi-layer encoder module is based on a multi-head attention mechanism, a fully connected layer, a Dropout layer, and a layer standardization design for each encoder structure. The specific processing procedure is as follows: Word embedding vectors based on multi-head attention mechanism The feature processing makes the encoder able to capture key features in the engineering cost data from different angles; Computations , and Matrix: in, , and These are query, key, and value functions, respectively. , and They represent the first time. The query, key, and value matrix under each attention head, and the total number of multi-head attention heads is... ,but ; After that, the multi-head attention feature is calculated : wherein, is a softmax function, is a normalization factor; Based on the feature , by splicing and processing the feature vectors of each head, the information of multiple viewing angles is integrated, and the calculation of the multi-head attention processing result is: wherein, is a feature processing vector of the th attention head, and ; is a channel concatenation operation; Then layer normalization processing is performed, and then a fully connected layer, a Dropout layer, and a residual connection are passed to obtain extracted features ; Finally, the feature processing result of the first layer encoder is obtained again by layer normalization ; The constructed encoder structure is stacked to form a 10-layer structure, where the input of each layer encoder is the output of the layer above it; the output of the last layer encoder... This represents the overall processing result of the multilayer encoder module; The multi-layer decoder module is based on a multi-head attention mechanism, a fully connected layer, a Dropout layer, and layer standardization design for each decoder structure. The specific processing procedure is as follows: The output of the multi-layer encoder module is input into a multi-layer decoder module As the to-be-decoded vector, the encoding vector is the processing result of the encoder layer corresponding to the layer decoder, the features of the encoder are directly introduced into the decoder through the jump connection, and the decoding engineering cost data is processed after the multi-head attention. The processing result is: wherein, is an encoding vector, and ; and are multi-head attention processing results of the encoding vector and the vector to be decoded, respectively. Will and The features are obtained by adding them together and then processing them using layer normalization. ; Then, the further extracted features are obtained through a fully connected layer, a Dropout layer and a residual connection : The feature processing result of the first layer decoder is obtained again by layer normalization ; The constructed decoder structure is stacked, having the same number of layers as the multi-layer encoder structure. The input vector to each subsequent decoder layer is the output of the previous layer, and the input encoded vector to each decoder layer is the output of the corresponding encoder layer. The output of the last decoder layer... This represents the overall processing result of the multi-layer decoder module; The word embedding layer is used to obtain word encoding vectors and positional encoding vectors, and generate word embedding results; the multi-layer encoder module is used to deeply extract hierarchical features of the data. Each encoder layer takes the output of the previous layer as input, enabling the model to capture more complex features and patterns; the multi-layer decoder module incorporates the features of the corresponding encoder layer into the input of each decoder layer through skip connections and multi-head attention, decoding and restoring complex features layer by layer. Each decoder layer further fuses the features of the encoder for feature extraction and optimization; the output layer is used to transform the representation generated by the decoder back into the probability distribution in the vocabulary. For the completed engineering cost analysis model, pre-training is first performed to obtain a preliminary professional scenario model that adapts to engineering cost scenarios. Then, fine-tuning and training are performed based on engineering cost problem data to obtain a professional engineering cost automatic extraction and analysis model. The specific process of the pre-training is as follows: Based on the collected engineering project texts , and , trained with an autoregressive language modeling method, i.e., predicting the next word given the preceding text in each segment ; and defining a pre-trained autoregressive loss function as: wherein, is the segment text corresponding to the autoregressive prediction number, denotes the current word, denotes the word before , and is a model parameter, which is obtained by random initialization; The loss value is calculated based on randomly initialized model parameters, and the Adam optimization algorithm is used to optimize the model parameters and update the gradients. wherein, denote the updated model parameters, is a learning rate; and setting adaptive learning rate for model training: setting lower learning rate for shallow layers, while higher learning rate for deep layers; After that, the parameter iteration update is performed, and when the preset maximum number of times is reached The pre-training is stopped, and the trained model is saved as a pre-trained model. The specific process of the fine-tuning training is as follows: The engineering cost problem As model input, and to generate corresponding information data The target training model, the corresponding fine-tuning training text generation loss function Design: in, and Representing text respectively and The corresponding word vectors, This represents the model parameters during fine-tuning training, which are initialized to the pre-trained model parameters. The training of shallow layer parameters is unfrozen layer by layer using an asymptotic fine-tuning strategy. The specific steps include: Model parameter initialization : The last layer of the model The layer network is unfrozen and fine-tuned through training and parameter updates to adapt it to specific engineering cost analysis tasks, thereby optimizing and updating the model parameters: wherein, denotes the layer updated model parameters, is the learning rate; the parameter update of the last layer is completed after iterations; Continue with the model The layer network is unfrozen and fine-tuning training and parameter updates continue; wherein, and are the parameters of the first layer network before and after the update, respectively, at this time ; continue to update the parameters after the number of iterations, and complete this parameter update. Continue thawing The training strategy involves progressively fine-tuning the network parameters layer by layer, starting from the first layer of the model. When the parameters of all layers are unfrozen, the model enters the global fine-tuning stage, at which time all parameters are uniformly optimized simultaneously, and the preset maximum number of fine-tuning training is reached The model fine-tuning training is stopped.

2. The large model-based automatic extraction and analysis method for engineering cost according to claim 1, characterized in that, The process of collecting the original dataset for pre-training the large-scale engineering cost analysis model includes: filtering out key information related to cost analysis from project documents through keyword matching, including quantitative clauses in the contract, budget details in the bidding documents, material specifications, material usage, and labor costs; at the same time, in order to fully enhance the automatic cost extraction and analysis capabilities by utilizing the large-scale model's ability to process related information, project scale, geographical location, and construction conditions are also included as raw text data.

3. The large model-based engineering cost automatic extraction and analysis method of claim 2, wherein, The specific process of labeling the original dataset is as follows: The system provides corresponding data annotations for different cost issues, including detailed information related to engineering costs. Based on the needs of engineering cost analysis, the data is annotated at multiple levels, with specific cost items annotated at a fine-grained level and the overall cost structure of the project annotated at a coarse-grained level. Each set of issues is labeled with the corresponding region, so that during the pre-training of the large model, the system can automatically adapt to the local units, cost structure, and cost categories for engineering projects in different regions.

4. The large model-based engineering cost automatic extraction and analysis method of claim 1, wherein: The word embedding layer includes word encoding and position encoding, and the specific processing procedure is as follows: First, the text is decomposed and encoded using the WordPiece word embedding method, and the word encoding vectors corresponding to the text data are obtained: in, The text data to be encoded. This is the encoded word vector. For word embedding functions; Secondly, the position code is calculated using sine and cosine functions, namely: wherein, denotes the position of each word in the text sequence, is an encoding dimension index, is a position encoding dimension, denotes the result of the position encoding; Finally, word encoding vectors and position encoding vectors with the same data dimensions are element-wise added to generate word embedding results .

5. The method for automatic extraction and analysis of engineering costs based on a large model as described in claim 1, characterized in that: After the multi-layer decoder obtains the final processing result, the output layer transforms the representation generated by the decoder back into the probability distribution in the vocabulary. The output layer consists of a fully connected layer and a softmax function, i.e.: wherein, is the final generated word vector output, denotes a softmax processing function.

Images