Unstructured data mapping method and system based on large language model and BIM model

Abstract

This invention discloses a method and system for mapping unstructured data based on large language models and BIM models, applicable to construction quality and safety management in building engineering. The method first establishes data standards and builds a BIM model with LOD400 accuracy; then, using OCR and large language model technology, it transforms messy unstructured data from the site, such as paper reports and images, into structured data; next, it extracts key indicators, including sub-items and construction locations, and automatically maps the data to specific components of the BIM model through these key indicators, ultimately achieving automatic data-model linkage. Compared to traditional methods relying on component IDs, this invention innovatively achieves automated processing and association of unstructured data through large language model technology, significantly improving the data-model mapping efficiency in digital twin technology.

Description

Technical Field

[0001] This invention relates to the field of building digital twin technology, and in particular to a method and system for mapping unstructured data based on large language models and BIM models. Background Technology

[0002] Digital twins refer to the simulation process that fully utilizes data from physical models, sensor updates, and operational history to integrate multiple disciplines, physical quantities, scales, and probabilities, completing a mapping in virtual space to reflect the entire lifecycle of the corresponding physical equipment. A digital twin system, based on data-driven dynamic iteration of various elements between the target entity and the digital entity, typically consists of the target entity, the digital entity, the data connection between them, and the models, data, and interfaces involved in the data connection process. It mainly includes the target entity, twin interaction, the digital entity, and service applications. In the practical application of digital twins in the construction industry, data mapping is the core bridge connecting the physical and digital worlds. However, this stage currently faces many deep-seated technical and management challenges, directly hindering the leap from "visual display" to "intelligent decision-making" in digital twins. The ability to automatically attach data to corresponding components in the BIM model is an effective means to achieve true digital twins.

[0003] Currently, construction quality and safety management in building engineering relies heavily on a large amount of unstructured data generated on-site, such as paper inspection reports, handwritten records, and on-site images. Traditional digital twin technology often associates physical devices with data on the BIM model through preset component IDs, making it difficult to directly process such messy unstructured information. This results in data entry relying mainly on manual labor, which is inefficient and prone to errors, creating "information silos." Due to the lack of unified standards for multi-source heterogeneous data (IoT, point cloud, video) on the physical side, it is difficult to automatically align BIM model components, and unique identifiers are easily lost. Furthermore, the virtual model updates are often slower than the physical site due to network latency, computing power limitations, and indoor positioning errors. Simultaneously, the lack of a dynamic adaptive mechanism makes it difficult for the model to automatically evolve with the project progress. At present, digital twins cannot fully solve problems such as spatial mapping and information redundancy. These problems cause digital twins to largely become static displays, failing to truly form a closed loop of "perception-analysis-decision-control," thus restricting the release of their intelligent decision-making value. Summary of the Invention

[0004] The problem to be solved by this invention is to provide a method and system for unstructured data mapping based on large language models and BIM models, which can improve the efficiency of data linking and effectively reduce the technical understanding threshold and application difficulty for industry practitioners.

[0005] This invention adopts the following technical solution: a method for mapping unstructured data based on a large language model and a BIM model, comprising the following steps:

[0006] S01. Establish data standards for construction quality and safety in building engineering projects;

[0007] S02. Based on the data standard, establish a building engineering BIM model with LOD 400 component-level accuracy, including grid lines, floors, component types, component size information, and divide it into sub-items of the project;

[0008] S03. Based on OCR methods and large language models, structured processing of construction quality and safety data at the construction site is performed.

[0009] S04. Extract key indicators based on the structured construction quality and safety data;

[0010] S05. Based on the key indicators, call the BIM engine to obtain the grid and floor information in the BIM model, and map the construction quality and safety data to the components of the building engineering BIM model.

[0011] S06. The BIM model of the building project will visualize the construction quality and safety data that are attached to it.

[0012] As a preferred option, in step S01, the data standards for construction quality and safety of building engineering include project name, project progress, quality management objectives, material properties, dimensional properties, sub-items of engineering, construction location, construction time, inspection object, inspection time, inspection conclusion, acceptance object, acceptance time, and acceptance conclusion.

[0013] As a preferred option, in step S03, the construction quality and safety data at the construction site are processed in a structured manner. The specific steps are as follows:

[0014] S301. Image preprocessing is performed using the global thresholding method;

[0015] S302. Perform text detection and localization, including: determining whether the image belongs to text, predicting the position of the text box, and confirming the detected text.

[0016] S303. Convert paper images into machine-readable structured text using OCR methods, perform character content recognition, correct the recognized text content, and output clean text.

[0017] S304. Embed the identified text at specific locations;

[0018] S305. Use a self-attention mechanism to understand the text content and obtain industry standard terminology for use in BIM model integration.

[0019] S306. Using a large language model, input the industry standard terms and generate structured data according to the objective function.

[0020] As a preferred option, in step S04, based on the generated structured data, key indicator data is extracted according to the indicators in the data standard for construction quality and safety. The key indicators include sub-items of the project and construction locations. The extracted key indicators are mapped to the corresponding positions in the building engineering BIM model through the construction locations.

[0021] As a preferred option, in step S05, the construction quality and safety data are mapped to the components of the building engineering BIM model, as follows:

[0022] S501. Input the key indicators of the construction location in the structured construction quality and safety data, which are used to indicate the location of the grid.

[0023] S502. Based on the scope of the construction site, find the geometric coordinates of that scope in the BIM model, call the BIM engine to query all components within that scope, and output the Node ID of all components.

[0024] S503. Automatically input other indicators from the structured construction quality and safety data, including material properties, dimensional properties, sub-items, construction time, inspection time, inspection conclusion, acceptance time and acceptance conclusion, into the corresponding components of the BIM model, thereby realizing the automatic mapping of unstructured data on construction quality and safety of building projects and expressing it at the component level on the BIM model.

[0025] The present invention also provides: an unstructured data mapping system based on a large language model and a BIM model, comprising:

[0026] The data standards development module is used to develop data standards for construction quality and safety in building engineering projects.

[0027] The BIM model creation module is used to create BIM models of building projects with LOD 400 component-level precision.

[0028] The OCR and large language model calling module is used to perform structured processing of construction quality and safety data at the construction site based on OCR methods and large language models, and to extract key indicators based on the structured construction quality and safety data.

[0029] The data input and BIM engine calling module is used to obtain the grid and floor information in the BIM model by calling the BIM engine according to the key indicators, and to link and map the construction quality and safety data to the components of the building engineering BIM model.

[0030] The data output module visualizes the construction quality and safety data linked to the building engineering BIM model.

[0031] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0032] 1. The unstructured data mapping method of this invention uses large language model technology to structure the unstructured data of construction quality and safety at the construction site. Based on the structured on-site construction quality and safety data, key indicators, sub-items of the project, construction location (location on the grid), and other necessary quality and safety data are extracted. The extracted data is then mapped onto the BIM model through sub-items of the project and construction location (location on the grid). This innovatively achieves automated processing and association of unstructured data through large language model technology.

[0033] 2. The invention selects two core indicators in the construction process, namely "sub-item project" and "construction location", as key semantic indexes for automatic association between data and BIM model by using OCR and large language model technology. This replaces the traditional method of relying on "component ID" as a unique identifier, which significantly improves the efficiency of data association, enhances the mapping efficiency between data and model in digital twin technology, and effectively reduces the technical understanding threshold and application difficulty for industry practitioners. Attached Figure Description

[0034] Figure 1 This is a flowchart of the unstructured data mapping method based on a large language model and a BIM model according to the present invention;

[0035] Figure 2 This is a schematic diagram of the unstructured data mapping system based on a large language model and a BIM model according to the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0037] Example 1

[0038] This paper presents a method for mapping unstructured data based on large language model and BIM model, which is applied to construction quality and safety management. The method uses large language model technology to structure the unstructured data of construction quality and safety at the construction site. Based on the structured on-site construction quality and safety data, key indicators, sub-items of the project, construction location (location on the grid line) and other necessary quality and safety data are extracted. The extracted data is then mapped onto the BIM model through the sub-items of the project and the construction location (location on the grid line).

[0039] The method flow in this embodiment is as follows: Figure 1 As shown, it includes the following steps:

[0040] Step 1: Establish data standards for construction quality and safety in building engineering projects;

[0041] Step 2: Create a BIM model of the building project with LOD 400 component-level precision, including information such as grid lines, floors, component types, and component dimensions, and divide it into sub-projects;

[0042] Step 3: Based on OCR and large language model technology, perform semantic recognition and structuring processing on unstructured / semi-structured construction quality and safety data;

[0043] Step 4: Extract key indicators based on the structured construction quality and safety data, including sub-items of the project and construction location (location on the grid line).

[0044] Step 5: Based on key indicators, map construction quality and safety data to the components of the BIM model using the BIM engine;

[0045] Step 6: The BIM model will perform statistical analysis and model rendering on the attached construction quality and safety data, and finally present it through visualization.

[0046] Specifically, the data standards for construction quality and safety include project name, project progress, quality management objectives, material properties, dimensional properties, sub-items of the project, construction location (location on the grid line), construction time, inspection objects, inspection time, inspection conclusions, acceptance objects, acceptance time, and acceptance conclusions.

[0047] The structuring of construction quality and safety data first involves converting paper images into machine-readable structured text using OCR technology and a large language model. The large language model must be deployed on a local server to ensure data security. The specific steps are as follows:

[0048] Step 3.1: Image preprocessing uses the global thresholding method (Otsu), with the following formula:

[0049] ;

[0050] in, These are the grayscale values ​​of the original image pixels. This is the optimal threshold calculated by the Otsu algorithm. It is the pixel value after binarization.

[0051] Step 3.2: Perform text detection and localization.

[0052] First, determine whether the image belongs to text by calculating the confidence prediction function:

[0053] ;

[0054] in, The feature map output by the CNN is in The feature vector at that location, , It is the learnable weights and biases for confidence prediction. It is an activation function that represents the probability that a pixel is text.

[0055] Then, the position of the text box is predicted using the following formula:

[0056] ;

[0057] in, , These are the center coordinates of the text box. , These are the width and height of the text box. Rotate the text box by the specified angle. It is the convolutional layer mapping function for the geometric prediction branch.

[0058] Next, the detected text is confirmed, and the confidence loss is calculated accordingly. and geometric loss The calculation is performed using the following formula:

[0059] ;

[0060] in, A binary cross-entropy loss is used to extract text / non-text pixels. A smoothed L1 loss method is used to constrain the deviation between the predicted bounding box and the ground truth bounding box. It is a balancing weight.

[0061] Step 3.3: Perform character content recognition, calculated as follows:

[0062] ;

[0063] in, It is the output character probability sequence. Is the target character sequence and Aligned path, It is a negative log-likelihood loss.

[0064] Step 3.4: Correct the recognized text content:

[0065] First, based on the initial text sequence Generate possible error candidate words :

[0066] ;

[0067] in, It is the edit distance; Candidate words represent all possible replacements. The word; It is the i-th text sequence. It is a threshold.

[0068] Then, select the most fluent text sequence based on the perplexity of the language. The formula for calculating perplexity is:

[0069] ;

[0070] in, It is the most fluent text sequence. It is a text sequence The level of confusion, It is the character conditional probability predicted by the large language model.

[0071] Step 3.5: Embed the recognized text at specific locations using the following formula:

[0072] ;

[0073] in, It is the first A one-hot vector of each token. It is a vocabulary embedding matrix. It is a positional encoding vector. It is a positional encoding matrix.

[0074] Step 3.6: Understand the text content through a self-attention mechanism. The formula is:

[0075] ; ;

[0076] in, It is a query, key, and value matrix. For the output of the self-attention mechanism, The function is a concatenation function, used to concatenate... The outputs of each attention head are concatenated along the feature dimension to form a long vector; To output the weight matrix, It's the number of heads that attract attention. ; The function is a single-head attention calculation function. For normalization function, These are learnable parameters. It is the dimension of the key vector.

[0077] Step 3.7: Deploy the large language model on a local server and generate structured data according to the objective function; the objective function formula is:

[0078] ;

[0079] in, It is candidate structured data. For the generated structured data, It is clean text output by OCR. It is a constraint of target structure. It is conditional probability. It is used to output the maximum conditional probability, i.e., the optimal solution.

[0080] Furthermore, after structuring the unstructured construction quality and safety data of the construction site through OCR and large model technology, the key indicators are extracted according to the indicators in the data standard for construction quality and safety, and the extracted key indicators can be mapped to the corresponding positions in the BIM model through the construction location (the location of the axis grid).

[0081] The BIM model must include key information that is the same as or similar to the description of construction quality and safety on the construction site, such as grid lines, floors, component types, and component dimensions, so that the BIM model can automatically obtain the location of model components through grid information.

[0082] Furthermore, the grid and floor information in the BIM model are obtained by calling a domestic BIM engine, as follows:

[0083] First, input the key indicator "construction location (location on the grid)" from the structured construction quality and safety data, and then find the geometric coordinates of that location in the BIM model based on the scope of the construction location.

[0084] Then, the BIM engine is called to query all components within the scope and output the Node ID of all components. The machine can automatically input other indicators in the structured construction quality and safety data, such as material properties, dimensional properties, sub-items, construction time, inspection time, inspection conclusion, acceptance time, and acceptance conclusion, into the corresponding components of the BIM model. This enables the automatic mapping of unstructured data on construction quality and safety of building projects and their component-level expression on the BIM model.

[0085] Example 2

[0086] This embodiment provides an unstructured data mapping system based on a large language model and a BIM model, used to execute the method of Embodiment 1.

[0087] The system in this embodiment includes:

[0088] The data standard setting module is used to perform step 1, which involves setting data standards for construction quality and safety in building engineering.

[0089] The BIM model creation module is used to perform step 2, creating a building engineering BIM model with LOD 400 component-level precision;

[0090] The OCR and large language model calling module is used to execute steps 3 to 4. First, it performs the structuring of construction quality and safety data based on OCR and large language model technology. Then, it extracts key indicators of construction location (location on the grid line) based on the structured construction quality and safety data.

[0091] The data input and BIM engine calling module is used to execute step 5, which maps construction quality and safety data to components in the BIM model based on key indicators.

[0092] The data output module is used to execute step 6, which visualizes the attached construction quality and safety data through the BIM model.

[0093] In this embodiment, the unstructured data automatic mapping algorithm is first developed using the Python programming language; then, a backend application is written in Java to call the above-mentioned structured data automatic mapping method and related business data; finally, the visualization of the data mapping results is completed using the frontend Vue technology.

[0094] Specifically, such as Figure 2 As shown, firstly, data on construction quality and safety are obtained, including project name, project progress, quality management objectives, material properties, dimensional properties, sub-items of the project, construction location (location on the grid line), construction time, inspection objects, inspection time, inspection conclusions, acceptance objects, acceptance time, and acceptance conclusions.

[0095] Then, the unstructured or semi-structured data is structured and processed using the method of this invention to extract key indicators, including sub-items of the project and construction locations.

[0096] Next, the data is automatically linked to the model through key indicators, and then rendered and presented by attaching it to the BIM model, ultimately realizing the automatic mapping of construction quality and safety data, and further promoting the development of digital twins in the construction field.

[0097] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for mapping unstructured data based on a large language model and a BIM model, characterized in that, Includes the following steps: S01. Establish data standards for construction quality and safety in building engineering projects; S02. Based on the data standard, establish a building engineering BIM model with LOD 400 component-level accuracy, including grid lines, floors, component types, component size information, and divide it into sub-items of the project; S03. Based on OCR methods and large language models, structured processing of construction quality and safety data at the construction site is performed. S04. Extract key indicators based on the structured construction quality and safety data; S05. Based on the key indicators, call the BIM engine to obtain the grid and floor information in the BIM model, and map the construction quality and safety data to the components of the building engineering BIM model. S06. The BIM model of the building project will visualize the construction quality and safety data that are attached to it.

2. The unstructured data mapping method according to claim 1, characterized in that, In step S01, the data standards for construction quality and safety of building engineering include project name, project progress, quality management objectives, material properties, dimensional properties, sub-items of engineering, construction location, construction time, inspection object, inspection time, inspection conclusion, acceptance object, acceptance time, and acceptance conclusion.

3. The unstructured data mapping method according to claim 1, characterized in that, In step S03, the construction quality and safety data at the construction site are processed in a structured manner. The specific steps are as follows: S301. Image preprocessing is performed using the global thresholding method, with the following formula: ； in, These are the grayscale values ​​of the original image pixels. It is the optimal threshold calculated using the global thresholding method. These are the pixel values ​​after binarization; S302. Perform text detection and localization, including: determining whether the image belongs to text, predicting the position of the text box, and confirming the detected text. S303. Convert paper images into machine-readable structured text using OCR methods, perform character content recognition, correct the recognized text content, and output clean text. ; S304. Embed the identified text at specific locations using the following formula: ； in, It is the first A one-hot vector of each token. It is a vocabulary embedding matrix. It is a positional encoding vector. It is a positional encoding matrix; S305. Use a self-attention mechanism to understand the text content and obtain industry standard terminology for use in BIM model integration. S306. Using a large language model, input the industry standard terms and generate structured data according to the objective function.

4. The unstructured data mapping method according to claim 3, characterized in that, In step S302, it is determined whether the image belongs to text. This is done by calculating the confidence prediction function, as shown in the formula: ； in, The feature map output by the CNN is in The feature vector at that location, , It is the learnable weights and biases for confidence prediction. It is an activation function that represents the probability that a pixel is text; The formula for predicting the position of a text box is: ； in, , These are the center coordinates of the text box. , These are the width and height of the text box. Rotate the text box by the specified angle. It is the convolutional layer mapping function for the geometric prediction branch; The detected text is verified, and the confidence loss is calculated accordingly. and geometric loss The calculation formula is as follows: ； in, A binary cross-entropy loss is used to extract text / non-text pixels. A smoothed L1 loss method is used to constrain the deviation between the predicted bounding box and the ground truth bounding box. It is a balancing weight.

5. The unstructured data mapping method according to claim 3, characterized in that, In step S303, the character content recognition formula is as follows: ； in, It is the output character probability sequence. Is the target character sequence and Aligned path, The loss is negative log-likelihood. The correction of the identified text content includes: based on the initial text sequence Generate possible error candidate words ; Select the most fluent text sequence based on the perplexity of the language. ; in, It is the edit distance; Candidate words represent all possible replacements. The word; It is the i-th text sequence. It is a threshold; It is the most fluent text sequence. It is a text sequence The level of confusion; The perplexity is calculated using the following formula: ； in, It is the character conditional probability predicted by the large language model.

6. The unstructured data mapping method according to claim 3, characterized in that, In step S305, the text content is understood using a self-attention mechanism, with the following formula: ； ； in, It is a query, key, and value matrix. For the output of the self-attention mechanism, The function is a concatenation function, used to concatenate... The outputs of each attention head are concatenated along the feature dimension to form a long vector; To output the weight matrix, It's the number of heads that attract attention. ; The function is a single-head attention calculation function. For normalization function, These are learnable parameters. It is the dimension of the key vector.

7. The unstructured data mapping method according to claim 3, characterized in that, In step S306, the large language model is deployed on a local server, and the objective function formula is: ； in, It is candidate structured data. For the generated structured data, It is clean text output by OCR. It is a constraint of target structure. It is conditional probability. It is used to output the maximum conditional probability, i.e., the optimal solution.

8. The unstructured data mapping method according to claim 3, characterized in that, In step S04, based on the generated structured data, key indicator data is extracted according to the indicators in the data standard for construction quality and safety. The key indicators include sub-items of the project and construction locations. The extracted key indicators are mapped to the corresponding positions in the building engineering BIM model through the construction locations.

9. The unstructured data mapping method according to claim 8, characterized in that, In step S05, the construction quality and safety data are mapped to the components of the building engineering BIM model, as follows: S501. Input the key indicators of the construction location in the structured construction quality and safety data, which are used to indicate the location of the grid. S502. Based on the scope of the construction site, find the geometric coordinates of that scope in the BIM model, call the BIM engine to query all components within that scope, and output the Node ID of all components. S503. Automatically input other indicators from the structured construction quality and safety data, including material properties, dimensional properties, sub-items, construction time, inspection time, inspection conclusion, acceptance time and acceptance conclusion, into the corresponding components of the BIM model, thereby realizing the automatic mapping of unstructured data on construction quality and safety of building projects and expressing it at the component level on the BIM model.

10. An unstructured data mapping system based on a large language model and a BIM model, characterized in that, include: The data standards development module is used to develop data standards for construction quality and safety in building engineering projects. The BIM model creation module is used to create BIM models of building projects with LOD 400 component-level precision. The OCR and large language model calling module is used to perform structured processing of construction quality and safety data at the construction site based on OCR methods and large language models, and to extract key indicators based on the structured construction quality and safety data. The data input and BIM engine calling module is used to obtain the grid and floor information in the BIM model by calling the BIM engine according to the key indicators, and to link and map the construction quality and safety data to the components of the building engineering BIM model. The data output module visualizes the construction quality and safety data linked to the building engineering BIM model.

Images