Automatic engineering cost extraction and analysis method based on large model

Abstract

The invention provides an automatic engineering cost extraction and analysis method based on a large model, and belongs to the technical field of engineering cost data analysis. The method comprises the following steps: firstly, obtaining a to-be-analyzed engineering project text, inputting the engineering project text into a constructed and trained engineering cost analysis large model, and inputting a specific cost problem; the large engineering cost analysis model processes and analyzes an engineering project text to obtain each item of cost information of a corresponding project, and then performs corresponding reply generation on a specifically input cost problem to realize automatic extraction and analysis of the engineering cost; the large project cost analysis model comprises a word embedding layer, a multi-layer encoder module, a multi-layer decoder module and an output layer; corresponding layers of the multi-layer encoder module and the multi-layer decoder module form a U-Net architecture through jump connection; the method can automatically adapt to different data formats and complex scenes, manual design of rules and templates is not needed, and accurate analysis can be carried out when the specific cost problem is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of engineering cost data analysis, and in particular relates to an automatic extraction and analysis method of engineering cost based on a large model. Background Art

[0002] As the scale of infrastructure construction and engineering projects continues to expand, the complexity and diversity of engineering projects continue to increase. These projects involve a lot of resource allocation, cost control and schedule management, which puts higher requirements on the cost management of the project. Therefore, in the current engineering construction process, accurate cost management and control are important factors to ensure the success of the project. However, traditional engineering cost management methods mainly rely on manual experience and manual operations, which are difficult to handle the complex and changeable data in modern engineering, resulting in inaccurate cost estimates, insufficient information updates, and difficult scientific decision-making. These problems not only increase the risk of project overruns and schedule delays, but also affect the overall benefits of the project. Therefore, the research and development of automated engineering cost extraction and analysis technology can significantly improve work efficiency and data accuracy, reduce errors in manual operations, and thus reduce the risk of cost overruns in engineering projects. In addition, the application of this technology can provide modern engineering projects with more efficient, accurate and scientific cost management tools, and improve the overall management level and economic benefits of the project.

[0003] Existing automatic extraction and analysis methods for construction cost usually rely on rule-based cost analysis systems, machine learning-based analysis methods, and deep learning-based cost analysis methods. These methods can realize the automatic processing and analysis of construction cost data to a certain extent, but they still have some key shortcomings.

[0004] Rule-based cost analysis system: The rule-based cost analysis system extracts engineering cost information through predefined rules and templates, and relies on the knowledge of domain experts for design and maintenance. However, this approach has great limitations when dealing with different data formats and complex scenarios, and has poor flexibility and versatility. It also has high maintenance costs and is difficult to handle complex or unstructured data.

[0005] Machine learning-based analysis methods: Traditional machine learning algorithms, such as decision trees and support vector machines, are trained with hand-crafted features to predict and classify construction cost data. Although these methods perform well in simple tasks, their feature extraction process relies on the designer's experience, making it difficult to capture the deep relationships of complex data, and their scalability and generalization capabilities are weak when faced with large-scale and diverse data.

[0006] Cost analysis method based on deep learning: This method uses a complex neural network model to automatically extract features from engineering project data, and to a certain extent improves the accuracy and efficiency of cost analysis. However, it may not be able to adapt or migrate effectively when dealing with different types of engineering data or cross-domain applications, resulting in limited applicability in complex practical application environments, and may need to be adjusted or retrained for each specific task to meet specific needs. Summary of the invention

[0007] In view of the above problems, the present invention provides a method for automatically extracting and analyzing engineering cost based on a large model, which includes the following processes: Acquire a text of a project to be analyzed, wherein the text of the project contains key information related to cost analysis; Input the engineering project text into the constructed and trained engineering cost analysis model, and input specific cost issues; the engineering cost analysis model processes and analyzes the engineering project text to obtain various cost information of the corresponding project, and then generates corresponding answers to the specific input cost issues to achieve automatic extraction and analysis of engineering costs; The large model for engineering cost analysis includes a word embedding layer, a multi-layer encoder module and a multi-layer decoder module based on a multi-head attention mechanism, and an output layer; each corresponding layer of the multi-layer encoder module and the multi-layer decoder module is connected by a jump connection to form a U-Net architecture; The word embedding layer is used to obtain the word encoding direction and position encoding vector, and generate the word embedding result; the multi-layer encoder module is used to deeply extract the hierarchical features of the data, and each layer of the encoder takes the output of the previous layer as input, so that the model can capture more complex features and patterns; the multi-layer decoder module integrates the features of the corresponding layer of the encoder into the input of each layer of the decoder through jump connections and multi-head attention, decodes and restores the complex features layer by layer, and each layer of the decoder further integrates the features of the encoder to perform feature extraction and optimization; the output layer is used to convert the representation generated by the decoder back to the probability distribution in the vocabulary.

[0008] Preferably, the process of collecting the original data set for pre-training the large model for engineering cost analysis includes: filtering out key information related to cost analysis in project documents through keyword matching, including quantitative terms in contracts, budget details in bidding documents, material specifications, material usage, and labor costs; at the same time, in order to utilize the large model's ability to process contextual information to fully enhance automatic cost extraction and analysis capabilities, contextual information such as project scale, geographic location, and construction conditions are included as original text data.

[0009] Preferably, the specific process of data labeling the original data set is: Corresponding data annotation is performed for different cost issues, including detailed information related to the project cost. According to the needs of project cost analysis, the data is annotated at multiple levels, with specific cost items annotated at the fine-grained annotation level and the cost structure of the entire project annotated at the coarse-grained annotation level. The corresponding region is labeled for each group of issues, so that during the pre-training process of the large model, the local units, cost structure and cost categories can be automatically adapted for engineering projects in different regions.

[0010] Preferably, the word embedding layer includes word encoding and position encoding, and the specific processing process is: First, the text is decomposed and encoded based on the WordPiece word embedding method, and the word encoding vector corresponding to the text data is obtained:

[0011] in, is the text data to be encoded, is the encoded word encoding vector, is the word embedding function; Secondly, the position code is calculated using the sine and cosine functions, namely:

[0012] in, Indicates the position of each word in the text sequence. is the encoding dimension index, is the dimension of the position encoding, Represents the result of position encoding; Finally, the word encoding vector with the same data dimension and the positional encoding vector Perform element-by-element addition to generate word embedding results .

[0013] Preferably, the multi-layer encoder module designs each encoder structure based on a multi-head attention mechanism, a fully connected layer, a Dropout layer, and layer standardization, and the specific processing process is: Word embedding vector based on multi-head attention mechanism Perform feature processing so that the encoder can capture key features in the engineering cost data from different angles; calculate , and matrix:

[0014] in, , and They are query, key and value functions respectively. , and Respectively expressed in The query, key and value matrices under the attention heads, the total number of multi-head attention heads is ,but ; After that, calculate the multi-head attention features :

[0015] in, is the softmax function, is the normalization factor; Feature-based By splicing and processing the feature vectors of each head, integrating information from multiple perspectives, and enhancing the diversity and expression ability of features, the multi-head attention processing results The calculation is:

[0016]

[0017] in, No. The feature processing vector of the attention head, and ; It is a channel splicing operation; Perform layer normalization to reduce the gradient vanishing problem of the model when processing complex cost data, namely:

[0018] in, is the standard feature after layer normalization, Normalize the function for the layer; Then, the fully connected layer, Dropout layer and residual connection are used to obtain further extracted features. :

[0019]

[0020] in, and Represent the fully connected layer and Dropout layer processing functions respectively; Represents the extracted encoder intermediate features; Finally, the feature processing result of the first layer encoder is obtained by layer normalization again :

[0021] The encoder structure is stacked to form a 10-layer structure, and the input of each layer of encoder is the output of the previous layer of encoder; the output of the last layer of encoder is It is the overall processing result of the multi-layer encoder module.

[0022] Preferably, the multi-layer decoder module designs each encoder structure based on a multi-head attention mechanism, a fully connected layer, a Dropout layer, and layer standardization, and the specific processing process is: The output of the multi-layer encoder module As the vector to be decoded, the encoding vector is the processing result of the encoder layer corresponding to the decoder of this layer. The features of the encoder are directly introduced into the decoder through the jump connection to more accurately decode the engineering cost data. The processing results after multi-head attention are:

[0023]

[0024] in, is the encoding vector, and ; and These are the multi-head attention processing results for the encoding vector and the vector to be decoded; Will and Add and process using layer normalization:

[0025] in, Normalize the processed features for the decoder layer; Further extracted features are obtained through fully connected layers, Dropout layers and residual connections :

[0026]

[0027] in, Represents the extracted decoder intermediate features; The feature processing results of the first layer decoder are obtained by layer normalization again. ,Right now:

[0028] The decoder structure constructed is stacked, with the same number of layers as the multi-layer encoder structure, and the input vector to be decoded of each subsequent decoder layer is the output of the previous decoder layer, and the input encoding vector of each decoder layer is the output result of the corresponding encoder layer; the output of the last decoder layer is It is the overall processing result of the multi-layer decoder module.

[0029] Preferably, the output layer converts the representation generated by the decoder back to the probability distribution in the vocabulary after the multi-layer decoder obtains the final processing result. The output layer is composed of a fully connected layer and a softmax function, that is:

[0030] in, is the final generated word vector output, Represents the softmax processing function.

[0031] Preferably, for the constructed engineering cost analysis large model, pre-training is first performed to obtain a preliminary professional scenario large model that adapts to the engineering cost scenario, and then fine-tuning training is performed based on the engineering cost problem data to obtain a professional engineering cost automatic extraction and analysis large model; wherein the fine-tuning training adopts an asymptotic fine-tuning strategy for training, which gradually adapts to the unique data characteristics of engineering cost by unfreezing the network parameters layer by layer, while retaining the general knowledge learned in the pre-training stage.

[0032] Preferably, based on the collected project engineering texts ,and , using the autoregressive language modeling method for training, that is, through each segment Predict the next word given the previous text in the text; and define the pre-trained autoregressive loss function for:

[0033] in, For the The number of autoregressive predictions corresponding to the segment text, Indicates the current word, Indicated in The previous word, are model parameters, which are obtained by random initialization; The loss value is calculated based on the randomly initialized model parameters, and the Adam optimization algorithm is used to optimize the model parameters and update the gradient:

[0034] in, represents the updated model parameters, is the learning rate; and an adaptive learning rate is set for model training: a lower learning rate is set for the shallow layer, and a higher learning rate is used for the deep layer to ensure the stability of the model training process; Then the parameters are iteratively updated. When the preset maximum number of times is reached Then stop pre-training and save the optimized trained model as the pre-training model. Preferably, the project cost issue As model input, and to generate corresponding information data Train the model for the target and fine-tune the training text to generate the loss function Designed for:

[0035] in, and Represents text and The corresponding word vector, Represents the model parameters during model fine-tuning training, which are initialized to the model parameters obtained by pre-training; The training of shallow parameters is unfrozen layer by layer using the gradual fine-tuning strategy. The specific steps include: Model parameter initialization :

[0036] in, Respectively represent the model from layer 1 to layer The network layer parameters of each network layer are initialized to the pre-trained model parameters, and all weight parameters are frozen; Unfreeze the last layer of the model network, perform fine-tuning training and parameter update, so as to first adapt to the specific engineering cost analysis task, and further optimize and update the model parameters:

[0037] in, Indicates The model parameters after layer update, is the learning rate, which is consistent with the pre-training process; cyclic update After iterations, the parameter update of the last layer is completed; Continue to unfreeze the penultimate layer of the model and continue fine-tuning training and parameter updates, namely:

[0038] in, and Respectively The parameters of the layer network before and after updating. ; Continue to update the loop After iterations, the parameter update is completed; Continue to thaw From layer to layer 1 of the model, the asymptotic fine-tuning training strategy of the model network parameters is finally completed layer by layer; When all the parameters of the layers are unfrozen, the model enters the global fine-tuning phase, where all parameters are optimized uniformly and the preset maximum number of fine-tuning training is reached. Then stop model fine-tuning training.

[0039] Compared with the prior art, the present invention has the following beneficial effects: (1) Efficient and flexible automatic analysis of construction cost: The present invention uses large model technology, WordPiece word embedding and multi-head attention mechanism to effectively process and analyze complex construction cost data. Compared with traditional analysis methods, the present invention can automatically adapt to different data formats and complex scenarios, and does not require manual design of rules and templates, which significantly improves the flexibility and adaptability of the system.

[0040] (2) Large-scale feature extraction of construction cost data: Based on a large-scale engineering project database, the present invention can deeply extract and analyze the actual cost information of a wide range of engineering projects. At the same time, the multi-level feature extraction method based on multi-layer encoders, multi-layer decoders and jump connections can better capture the deep relationship between complex data, thereby making full use of existing engineering project data to provide reliable construction cost analysis.

[0041] (3) A more specialized large model for cost analysis: Through large-scale pre-training and model fine-tuning training based on a progressive fine-tuning training strategy, the present invention enables the model to not only adapt to specific engineering scenarios, but also to perform accurate analysis when faced with specific cost issues; and the model's real-time feedback capability ensures rapid response and accurate analysis of new project data, thereby improving the real-time and reliability of engineering cost analysis.

[0042] (4) Improving the intelligence level of cost analysis: Compared with existing methods, the model of the present invention can automatically extract and analyze cost information through online language interaction, reducing the need for manual intervention while improving the level of automation, thereby making the extraction and analysis of project costs more intelligent and efficient.

[0043] (5) Efficient information fusion and transmission: The large model designed based on the U-Net architecture of the present invention uses the encoder-decoder structure and the jump connection therein to efficiently capture and fuse the different levels of information in the engineering cost text, from vocabulary to sentence to paragraph level features, so as to more comprehensively understand the content of the engineering cost text. Among them, the jump connection in U-Net allows low-level information to be directly transmitted to high-level information, which helps to restore more detailed text features during the decoding process, improve the accuracy of text extraction and analysis, and improve the accuracy of engineering cost analysis; through the stacking of multiple layers of encoders and decoders, the model can gradually extract and integrate deep-level features in the text. This hierarchical structure helps the model better understand the complex relationships and contextual information in the engineering cost text. Compared with the large model based on the Transformer architecture, the large model designed based on the U-Net architecture of the present invention adopts a more direct information transmission method in the encoding and decoding process, reduces unnecessary computing overhead, and has relatively less demand for training data. In the field of engineering cost, it is difficult to obtain high-quality labeled data. Therefore, compared with the large model based on the Transformer architecture, the large model proposed by the present invention is more suitable for automatic extraction and analysis of engineering cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall structural frame diagram of the large model for engineering cost analysis of the present invention.

[0045] Figure 2 This is a structural block diagram of the encoder model of the multi-layer encoder module of the present invention.

[0046] Figure 3 This is a structural block diagram of a decoder model of a multi-layer decoder module of the present invention.

[0047] Figure 4 This is a flowchart of the overall process of large model pre-training and fine-tuning training of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The overall process of the present invention is: Construction of engineering cost database: Construct engineering cost data based on historical engineering project documents and various material costs and cost data, and clean the original data and unify the format, and finally obtain an engineering cost database containing real project data; Design a large model for engineering cost analysis: Based on the WordPiece word embedding method and the multi-head attention mechanism, a large model for engineering cost analysis is constructed, which consists of a word embedding layer, a multi-layer encoder module, a multi-layer decoder module, and an output layer; Model pre-training and fine-tuning training based on the large model of engineering cost analysis: Use the constructed engineering cost database to perform engineering cost pre-training to obtain a large language model that adapts to the engineering scenario, and then perform fine-tuning training on engineering cost issues to obtain a professional large model for automatic extraction and analysis of engineering cost; Model deployment and optimization: Deploy the trained large model for automatic extraction and analysis of engineering cost to the background server, use online language interaction to realize the automatic cost extraction and analysis of engineering projects by the large model, and feed back the analysis results of the large model to the user in real time; at the same time, use new engineering project data to improve and fine-tune the large model, so as to continuously improve the real-time accuracy of the model and the reliability of engineering cost analysis, and ultimately meet the needs of actual application scenarios.

[0050] 1. Construction of Engineering Cost Training Dataset 1. Raw text data collection: In the raw data collection stage, key information related to cost analysis in project documents is screened out through keyword matching, including quantitative terms in contracts, budget details in bidding documents, material specifications, material usage, labor costs and other comprehensive text information. At the same time, in order to utilize the ability of large models to process contextual information to fully enhance the automatic cost extraction and analysis capabilities, contextual information such as project scale, geographic location, and construction conditions are included as raw text data, so that the collected text data not only covers cost details, but also includes background factors that affect these details. These related information will serve as auxiliary data to provide a more comprehensive perspective for subsequent analysis.

[0051] 2. Data cleaning and preprocessing: Filter out non-cost-related project descriptions, lengthy legal clauses, and technical parameters that are not related to construction costs in the original text data to reduce invalid data and improve the quality of the data set; in addition, convert imperial units to metric units, unify currency symbols, and handle possible time format differences to ensure data consistency. Then, convert the cleaned text data into UTF-8 encoded plain text format, and segment the data to obtain the common A text data about the project, namely: (1) in, For complete text data, For the The project engineering text, and ; 3. Data annotation: In order to realize automated engineering cost extraction and analysis, it is necessary to conduct extensive data annotation on cost data to meet engineering cost requirements; Specifically, for different cost issues, professional personnel are organized to perform corresponding data annotation. The issues include detailed information related to the project cost, such as: What is the budget for construction material requirements during the construction process? What is the budget for labor costs? What are the equipment rental and maintenance costs? When is the expected completion time of the project? In addition, according to the needs of project cost analysis, the data is labeled at multiple levels. For example: at the fine-grained labeling level, specific cost items are labeled, such as "labor costs" and "material costs"; at the coarse-grained labeling level, the cost structure of the entire project is labeled, such as "direct costs" and "indirect costs". This multi-level labeling strategy helps the model use labeled data more flexibly in different analysis tasks. In addition, the corresponding region is labeled for each set of questions, so that during the model training process, for engineering projects in different regions, the large model can automatically adapt to local units, cost structures, and cost categories; each set of questions Corresponding project information Construct a set of engineering cost data ; Data enhancement: The collected engineering cost data is augmented by rewording, replacing synonyms, adjusting the text structure, etc. For example, "material cost" is replaced with "material expense" or "material expenditure" to obtain diversified data with the same semantics but different expressions. This method can effectively improve the model's ability to understand different expressions to ensure the generalization ability of the model, and finally obtain Group project cost data: (2) in, Indicates Group construction cost data, and .

[0052] 2. Construction of a large model for project cost analysis The overall structure of the large model for engineering cost analysis is as follows Figure 1 As shown, it includes a word embedding layer, a multi-layer encoder module and a decoder module based on a multi-head attention mechanism, and an output layer. Each corresponding layer of the multi-layer encoder and decoder is connected through a jump connection to form a U-Net architecture. The output feature map of the encoder is directly spliced ​​into the input feature map of the corresponding decoder. The jump connection can alleviate the gradient disappearance problem and promote the effective transmission of the gradient in the deep network, thereby accelerating the convergence of the model; at the same time, it can also help retain the original input information, thereby retaining the details and local information of the encoder features during the decoding process, and ultimately improving the accuracy of the model in extracting and analyzing cost information.

[0053] 1. Word Embedding Layer The word embedding layer includes word encoding and position encoding. First, the present invention decomposes and encodes the text based on the WordPiece word embedding method, and obtains the word encoding vector corresponding to the text data: (3) in, is the text data to be encoded, is the encoded word encoding vector, is the word embedding function; Secondly, the position code is calculated using the sine and cosine functions, namely: (4) in, Indicates the position of each word in the text sequence. is the encoding dimension index, is the dimension of the position encoding, Represents the result of position encoding; Finally, the word encoding vector with the same data dimension and the positional encoding vector Perform element-by-element addition to generate word embedding results ,Right now: (5).

[0054] 2. Multi-layer encoder module The multi-layer encoder module designs each encoder structure based on the multi-head attention mechanism, fully connected layer, Dropout layer and layer standardization; the multi-layer encoder module can deeply extract the hierarchical features of the data. Each layer of the encoder takes the output of the previous layer as input, so that the model can capture more complex features and patterns, thereby improving the accuracy and reliability of engineering cost analysis. The model structure diagram of each encoder layer is as follows Figure 2 shown.

[0055] The specific data processing process of the multi-layer encoder module is: (1) Word embedding vector based on multi-head attention mechanism Perform feature processing so that the encoder can capture key features in the engineering cost data from different angles and improve the ability to understand complex information; calculate , and matrix: (6) in, , and They are query, key and value functions respectively. , and Respectively expressed in The query, key and value matrices under the attention heads, the total number of multi-head attention heads is ,but .

[0056] (2) After that, calculate the multi-head attention features : (7) in, is the softmax function, is the normalization factor; (3) Feature-based By concatenating and processing the feature vectors of each head, the encoder can integrate information from multiple perspectives and enhance the diversity and expressiveness of features. The result of multi-head attention processing is The calculation is: (8) (9) in, No. The feature processing vector of the attention head, and ; is the channel splicing operation; then the multi-head attention processing function is: (10) Among them, the multi-head attention processing function Specifically represents the processing process from step (1) to step (3); (4) Perform layer normalization to reduce the gradient vanishing problem of the model when processing complex cost data, namely: (11) in, is the standard feature after layer normalization, Normalize the function for the layer; (5) Then, the fully connected layer, the Dropout layer and the residual connection are used to obtain further extracted features. : (12) (13) in, and Respectively represent the fully connected layer and Dropout layer processing functions; Represents the extracted encoder intermediate features; (6) Finally, the feature processing results of the first layer encoder are obtained by layer normalization again : (14) Therefore, the overall process of feature processing by the encoder is: (15) in, represents the encoder layer processing function, which specifically is the processing process from step (1) to step (6); (7) Multi-layer encoder structure: Based on the constructed encoder structure, the encoder is stacked. In this embodiment, there are 10 layers in total, and the input of each layer of encoder is the output of the previous layer of encoder; therefore, the output of each layer of encoder is , , ..., , and the output of the last encoder layer It is the overall processing result of the multi-layer encoder module.

[0057] 3. Multi-layer decoder module Each encoder structure is also designed based on the multi-head attention mechanism, fully connected layer, Dropout layer and layer standardization. However, the features of the corresponding layer of the encoder are additionally integrated into the input of each decoder layer through skip connection and multi-head attention, so that the complex features are decoded and restored layer by layer. Each decoder layer further integrates the features of the encoder, and extracts and optimizes features through fully connected layers, Dropout layers and layer standardization. This structure can ensure the full utilization of information in the decoding process and improve the model's ability to understand and analyze engineering cost data. The model structure block diagram of each decoder layer is shown in the figure below. Figure 3 shown.

[0058] The specific data processing process of the multi-layer decoder module is: (1) In this embodiment, the output of the multi-layer encoder module As the vector to be decoded, the encoding vector is the processing result of the encoder layer corresponding to the decoder of this layer, that is, the first layer decoder corresponds to the processing result of the 10th layer encoder (the second layer decoder corresponds to the processing result of the 9th layer encoder, and so on). By directly introducing the features of the encoder into the decoder through the jump connection, the multi-layer decoder can decode the engineering cost data more accurately and ensure the coherence of information transmission. The processing results after multi-head attention are: (16) (17) in, is the encoding vector, and ; and These are the multi-head attention processing results for the encoding vector and the vector to be decoded; (2) and Add and process using layer normalization: (18) in, Normalize the processed features for the decoder layer; (3) Further extracted features are obtained through fully connected layers, dropout layers and residual connections : (19) (20) in, Represents the extracted decoder intermediate features; (4) The feature processing results of the first layer decoder are obtained again through layer normalization , which helps the model output more standardized features and ensures that the decoder maintains stability and accuracy during feature transfer at different levels, namely: (twenty one) Therefore, the overall process of feature processing by the encoder is: (twenty two) in, represents the decoder layer processing function, which specifically is the processing process from step (1) to step (4); (5) Multi-layer decoder structure: The decoder structure constructed is stacked, which has the same number of layers as the multi-layer encoder structure, that is, 10 layers, and the input vector to be decoded of each subsequent decoder layer is the output of the previous decoder layer, and the input encoding vector of each decoder layer is the output result of the corresponding encoder layer; therefore, the output of each layer of decoding is , , ..., , the output of the last decoder layer It is the overall processing result of the multi-layer decoder module.

[0059] 4. Output Layer After the multi-layer decoder obtains the final processing result, the representation generated by the decoder needs to be converted back to the probability distribution in the vocabulary. The output layer consists of a fully connected layer and a softmax function, that is: (twenty three) in, is the final generated word vector output, Represents the softmax processing function.

[0060] 3. Engineering Cost Analysis Large Model Training Based on the constructed large model for engineering cost analysis, the model first needs to be pre-trained to obtain a preliminary large model for professional scenarios that adapt to engineering cost scenarios, and then fine-tuned based on engineering cost problem data to obtain a large model for professional engineering cost automatic extraction and analysis. The fine-tuning training adopts an asymptotic fine-tuning strategy to gradually adapt to the unique data characteristics of engineering cost by unfreezing the network parameters layer by layer, while retaining the general knowledge learned in the pre-training stage, thereby improving the model's analysis accuracy for different engineering cost problems. In addition, this strategy can optimize the consumption of computing resources during the training process, making the application of the model in engineering cost analysis more efficient and accurate. The overall process is as follows: Figure 4 shown.

[0061] 1. Large model pre-training The text data of the project engineering that will be collected Used for pre-training of large models for engineering cost analysis; specifically, based on the collected project engineering texts ,and , using the autoregressive language modeling method for training, that is, through each segment Predict the next word given the previous text in the text; and define the pre-trained autoregressive loss function for: (twenty four) in, For the The number of autoregressive predictions corresponding to the segment text, Indicates the current word, Indicated in The previous word, are model parameters, which are obtained by random initialization; The loss value is calculated based on the randomly initialized model parameters, and the Adam optimization algorithm is used to optimize the model parameters and update the gradient: (25) in, represents the updated model parameters, is the learning rate; and an adaptive learning rate is set for model training: a lower learning rate is set for the shallow layer, and a higher learning rate is used for the deep layer to ensure the stability of the model training process; Then the parameters are iteratively updated. When the preset maximum number of times is reached Then stop pre-training and save the optimized trained model as the pre-training model. 2. Model fine-tuning training The collected project cost data Used for fine-tuning training of large models for engineering cost analysis, Indicates Group construction cost data, and , Specifically, the project cost issue As model input, and to generate corresponding information data The model is trained for the purpose of automatically generating cost information corresponding to various engineering cost issues. Therefore, the text generation loss function of the corresponding fine-tuning training is Designed for: (26) in, and Represents text and The corresponding word vector, Represents the model parameters during model fine-tuning training, which are initialized to the model parameters obtained by pre-training; Model fine-tuning training uses a gradual fine-tuning strategy to unfreeze shallow parameters layer by layer. The specific steps include: (1) Model parameter initialization : (27) in, Respectively represent the model from layer 1 to layer The network layer parameters of the layer, specifically, the parameters of each network layer are initialized to the pre-trained model parameters, and all weight parameters are frozen (remain unchanged); (2) The last layer of the model ( The network of the first layer (layer) is unfrozen, and fine-tuned and trained and parameter updated to first adapt to the specific engineering cost analysis task, and then further optimize and update the model parameters: (28) (3) Among them, Indicates The model parameters after layer update, is the learning rate, which is consistent with the pre-training process; cyclic update After iterations, the parameter update of the last layer is completed; (4) Continue to add the second to last layer of the model ( The network is unfrozen and fine-tuning training and parameter updates are continued, namely: (29) in, and Respectively The parameters of the layer network before and after updating. ; Continue to update the loop After iterations, the parameter update is completed; (5) Continue to thaw The strategy of asymptotic fine-tuning of the model network parameters is finally completed layer by layer. By gradually unfreezing and optimizing the model parameters, the model can more accurately adapt to the characteristics of the engineering cost data while retaining the pre-training knowledge, effectively avoiding overfitting, and ultimately improving the accuracy of the model's analysis on different engineering cost issues. In addition, this strategy can also optimize the resource utilization efficiency of the training process and reduce the training time and consumption of computing resources.

[0062] (6) When the parameters of all layers are unfrozen, the model enters the global fine-tuning stage, at which time all parameters are optimized uniformly and the preset maximum number of fine-tuning training is reached. Then stop the model fine-tuning training; save the fine-tuned trained model as the final large model for engineering cost analysis, and as the final deployment model for engineering cost analysis.

[0063] 4. Model Deployment and Application Large model deployment: Select appropriate server hardware configuration and deploy the trained engineering cost analysis large model to the backend server cluster; Input text: The user inputs the text of the engineering project to be analyzed into the engineering cost analysis model for extraction and analysis by the model, and inputs specific cost issues; Large model processing and analysis: First, process and analyze the project text to obtain the cost information of the corresponding project, and then generate corresponding answers to the specific input cost questions to achieve automatic extraction and analysis of project costs; Model optimization: Improve and fine-tune the large model in real time based on new project data information, so that the large model can be updated and optimized synchronously, ultimately ensuring the real-time accuracy and reliability of engineering cost analysis to meet the needs of actual application scenarios.

[0064] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0065] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for automatic extraction and analysis of engineering cost based on a large model, characterized in that: The process includes: Acquire a text of a project to be analyzed, wherein the text of the project contains key information related to cost analysis; Input the engineering project text into the constructed and trained engineering cost analysis model, and input specific cost issues; the engineering cost analysis model processes and analyzes the engineering project text to obtain various cost information of the corresponding project, and then generates corresponding answers to the specific input cost issues to achieve automatic extraction and analysis of engineering costs; The large model for engineering cost analysis includes a word embedding layer, a multi-layer encoder module and a multi-layer decoder module based on a multi-head attention mechanism, and an output layer; each corresponding layer of the multi-layer encoder module and the multi-layer decoder module is connected by a jump connection to form a U-Net architecture; The word embedding layer is used to obtain the word encoding direction and position encoding vector, and generate the word embedding result; the multi-layer encoder module is used to deeply extract the hierarchical features of the data, and each layer of the encoder takes the output of the previous layer as input, so that the model can capture more complex features and patterns; the multi-layer decoder module integrates the features of the corresponding layer of the encoder into the input of each layer of the decoder through jump connections and multi-head attention, decodes and restores the complex features layer by layer, and each layer of the decoder further integrates the features of the encoder to perform feature extraction and optimization; the output layer is used to convert the representation generated by the decoder back to the probability distribution in the vocabulary.

2. The method for automatically extracting and analyzing engineering cost based on a large model as claimed in claim 1, characterized in that: The process of collecting the original data set used to pre-train the large model for engineering cost analysis includes: filtering out key information related to cost analysis in project documents through keyword matching, including quantitative terms in contracts, budget details in bidding documents, material specifications, material usage, and labor costs; at the same time, in order to utilize the large model's ability to process contextual information to fully enhance the automatic cost extraction and analysis capabilities, contextual information such as project scale, geographic location, and construction conditions are included as original text data.

3. The method for automatic extraction and analysis of construction cost based on a large model as claimed in claim 2, characterized in that: The specific process of data labeling the original data set is as follows: Corresponding data annotation is performed for different cost issues, including detailed information related to the project cost. According to the needs of project cost analysis, the data is annotated at multiple levels, with specific cost items annotated at the fine-grained annotation level and the cost structure of the entire project annotated at the coarse-grained annotation level. The corresponding region is labeled for each group of issues, so that during the pre-training process of the large model, the local units, cost structure and cost categories can be automatically adapted for engineering projects in different regions.

4. The method for automatically extracting and analyzing construction costs based on a large model as claimed in claim 1, characterized in that: The word embedding layer includes word encoding and position encoding, and the specific processing process is as follows: First, the text is decomposed and encoded based on the WordPiece word embedding method, and the word encoding vector corresponding to the text data is obtained: in, is the text data to be encoded, is the encoded word encoding vector, is the word embedding function; Secondly, the position code is calculated using sine and cosine functions, namely: in, Indicates the position of each word in the text sequence. is the encoding dimension index, is the dimension of the position encoding, Represents the result of position encoding; Finally, the word encoding vector with the same data dimension and the positional encoding vector Perform element-by-element addition to generate word embedding results .

5. The method for automatically extracting and analyzing engineering cost based on a large model as claimed in claim 1, characterized in that: The multi-layer encoder module designs each encoder structure based on the multi-head attention mechanism, fully connected layer, Dropout layer and layer standardization. The specific processing process is as follows: Word embedding vector based on multi-head attention mechanism Perform feature processing so that the encoder can capture key features in the construction cost data from different angles; calculate , and matrix: in, , and They are query, key and value functions respectively. , and Respectively expressed in The query, key and value matrices under the attention heads, the total number of multi-head attention heads is ,but ; After that, calculate the multi-head attention features : in, is the softmax function, is the normalization factor; Feature-based By splicing and processing the feature vectors of each head, integrating information from multiple perspectives, and enhancing the diversity and expression ability of features, the multi-head attention processing results The calculation is: in, No. The feature processing vector of the attention head, and ; It is a channel splicing operation; Perform layer normalization to reduce the gradient vanishing problem of the model when processing complex cost data, namely: in, is the standard feature after layer normalization, Normalize the function for the layer; Then, the fully connected layer, Dropout layer and residual connection are used to obtain further extracted features. : in, and Represent the fully connected layer and Dropout layer processing functions respectively; Represents the extracted encoder intermediate features; Finally, the feature processing result of the first layer encoder is obtained by layer normalization again : The encoder structure is stacked to form a 10-layer structure, and the input of each layer of encoder is the output of the previous layer of encoder; the output of the last layer of encoder is It is the overall processing result of the multi-layer encoder module.

6. The method for automatically extracting and analyzing construction costs based on a large model as claimed in claim 5, characterized in that: The multi-layer decoder module designs each encoder structure based on the multi-head attention mechanism, fully connected layer, Dropout layer and layer standardization. The specific processing process is as follows: The output of the multi-layer encoder module As the vector to be decoded, the encoding vector is the processing result of the encoder layer corresponding to the decoder of this layer. The features of the encoder are directly introduced into the decoder through the jump connection to more accurately decode the engineering cost data. The processing results after multi-head attention are: in, is the encoding vector, and ; and These are the multi-head attention processing results for the encoding vector and the vector to be decoded; Will and Add and process using layer normalization: in, Normalize the processed features for the decoder layer; Further extracted features are obtained through fully connected layers, Dropout layers and residual connections : in, Represents the extracted decoder intermediate features; The feature processing results of the first layer decoder are obtained by layer normalization again. ,Right now: The decoder structure constructed is stacked, with the same number of layers as the multi-layer encoder structure, and the input vector to be decoded of each subsequent decoder layer is the output of the previous decoder layer, and the input encoding vector of each decoder layer is the output result of the corresponding encoder layer; the output of the last decoder layer is It is the overall processing result of the multi-layer decoder module.

7. The method for automatically extracting and analyzing construction costs based on a large model as claimed in claim 1, characterized in that: After the multi-layer decoder obtains the final processing result, the output layer converts the representation generated by the decoder back to the probability distribution in the vocabulary. The output layer is composed of a fully connected layer and a softmax function, that is: in, is the final generated word vector output, Represents the softmax processing function.

8. The method for automatically extracting and analyzing engineering cost based on a large model as claimed in claim 1, characterized in that: For the completed engineering cost analysis model, pre-training is first performed to obtain a preliminary professional scenario model that adapts to the engineering cost scenario, and then fine-tuning training is performed based on engineering cost problem data to obtain a professional engineering cost automatic extraction and analysis model; The fine-tuning training adopts an asymptotic fine-tuning strategy, which gradually adapts to the unique data characteristics of engineering cost by unfreezing the network parameters layer by layer, while retaining the general knowledge learned in the pre-training stage.

9. The method for automatically extracting and analyzing engineering cost based on a large model as claimed in claim 8, characterized in that: The specific process of the pre-training is: Based on the collected project engineering text ,and , using the autoregressive language modeling method for training, that is, through each segment Predict the next word given the previous text in the text; and define the pre-trained autoregressive loss function for: in, For the The number of autoregressive predictions corresponding to the segment text, Indicates the current word. Indicated in The previous word, are model parameters, which are obtained by random initialization; The loss value is calculated based on the randomly initialized model parameters, and the Adam optimization algorithm is used to optimize the model parameters and update the gradient: in, represents the updated model parameters, is the learning rate; and an adaptive learning rate is set for model training: a lower learning rate is set for the shallow layer, and a higher learning rate is used for the deep layer to ensure the stability of the model training process; Then the parameters are iteratively updated. When the preset maximum number of times is reached Then stop pre-training and save the optimized trained model as the pre-training model.

10. The method for automatically extracting and analyzing engineering cost based on a large model as claimed in claim 8, characterized in that: The specific process of the fine-tuning training is as follows: The project cost issue As model input, and to generate corresponding information data Train the model for the target and fine-tune the training text to generate the loss function Designed for: in, and Respectively represent text and The corresponding word vector, Represents the model parameters during model fine-tuning training, which are initialized to the model parameters obtained by pre-training; The training of shallow parameters is unfrozen layer by layer using the gradual fine-tuning strategy. The specific steps include: Model parameter initialization : in, Respectively represent the model from layer 1 to layer The network layer parameters of each network layer are initialized to the pre-trained model parameters, and all weight parameters are frozen; Unfreeze the last layer of the model network, perform fine-tuning training and parameter update, so as to first adapt to the specific engineering cost analysis task, and further optimize and update the model parameters: in, Indicates The model parameters after layer update, is the learning rate, which is consistent with the pre-training process; cyclic update After iterations, the parameter update of the last layer is completed; Continue to unfreeze the penultimate layer of the model and continue fine-tuning training and parameter updates, namely: in, and Respectively The parameters of the layer network before and after updating. ; Continue to update the loop After iterations, the parameter update is completed; Continue to thaw From layer to layer 1 of the model, the asymptotic fine-tuning training strategy of the model network parameters is finally completed layer by layer; When all the parameters of the layers are unfrozen, the model enters the global fine-tuning phase, where all parameters are optimized uniformly and the preset maximum number of fine-tuning training is reached. Then stop model fine-tuning training.

Images