Model interaction and data fusion method based on AI agent and multi-modal deep learning

Abstract

This invention discloses a model interaction and data fusion method based on AI intelligent agents and multimodal deep learning, relating to the field of BIM management technology in building engineering. This method is implemented through a multi-layered architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer. It acquires on-site data through various triggering methods in the multimodal input layer; parses the on-site data using the distributed AI parsing layer; fuses multi-source business data through the multi-source data fusion layer, constructing a BIM knowledge graph with component IDs as core nodes; performs basic interaction with the on-site model, model analysis, on-site linkage, and real-time collaborative interaction across multiple terminals through the lightweight model interaction layer; and achieves intelligent BIM data query and multimodal deep interaction through the business scenario linkage layer.

Description

Technical Field

[0001] This invention relates to the field of BIM management technology for building engineering, and more specifically, to a method for model interaction and data fusion based on AI intelligent agents and multimodal deep learning. Background Technology

[0002] Existing BIM+AI interaction solutions have the following shortcomings:

[0003] 1. High operational threshold and limited interaction mode: Existing BIM+AI interaction solutions typically only support text or voice command-based interaction, without effectively integrating multimodal information such as images and on-site positioning. When on-site personnel have difficulty accurately describing the location or characteristics of components in words, the interaction cannot proceed smoothly. At the same time, complex model operations still rely on professional BIM engineers, making it difficult for front-line construction and inspection personnel to quickly get started.

[0004] 2. Deep disconnect between model and on-site management: Existing on-site problem control solutions mostly use BIM models as shallow spatial carriers for hazard marking, failing to achieve accurate binding of problems and components under multi-source data fusion; in environments without or with weak network, on-site personnel need to manually take photos and fill out forms, and then manually match the problems with BIM model components after returning to a network environment, which is cumbersome and prone to positioning errors.

[0005] 3. Lack of deep intelligent interaction between AI and BIM data: Existing engineering AI technologies mostly focus on hazard identification and graded rectification, and the knowledge source mainly relies on textual specification knowledge bases. They fail to deeply analyze and reconstruct geometric data, attribute data and spatial topological relationship data in BIM models; AI functions are limited to converting user commands into model operation APIs, and fail to provide proactive intelligent decision support based on integrated data.

[0006] 4. Disconnect between interaction results and business processes: The results generated after interaction with existing solutions (such as marking and positioning) usually cannot be directly transferred to the engineering management business system. They need to be manually entered and distributed again, resulting in low information flow efficiency and difficulty in forming a management closed loop of "problem discovery - task assignment - rectification and acceptance - data archiving".

[0007] To address the aforementioned issues, this invention proposes a model interaction and data fusion method based on AI intelligent agents and multimodal deep learning, in order to solve the technical problems of high interaction threshold, shallow data fusion, low level of intelligence, and disconnection from business processes in existing technologies. Summary of the Invention

[0008] The purpose of this invention is to provide a model interaction and data fusion method based on AI agents and multimodal deep learning, so as to solve the problems existing in the above-mentioned background technology.

[0009] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0010] This application provides a model interaction and data fusion method based on AI intelligent agents and multimodal deep learning. This method is implemented through a multi-layered architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer. The method includes the following specific steps:

[0011] On-site data is acquired through multiple triggering methods in the multimodal input layer, including text commands, voice commands, image recognition, on-site GPS positioning, and component scanning.

[0012] The distributed AI analysis layer is used to analyze the on-site data. The distributed AI analysis layer adopts a collaborative analysis system of lightweight AI at the edge and core AI in the cloud.

[0013] Based on the parsing results of the distributed AI parsing layer, the multi-source business data is fused through the multi-source data fusion layer, and a BIM knowledge graph is constructed with component ID as the core node.

[0014] Based on the BIM knowledge graph, a lightweight model interaction layer is used to enable basic interaction with the site model, model analysis and site linkage, and real-time collaborative interaction across multiple terminals.

[0015] Based on the interaction results of the lightweight model interaction layer, the business scenario linkage layer enables closed-loop linkage for quality rectification, technical briefing, progress control, and data archiving.

[0016] Based on the above technical solution, the present invention can be further improved as follows.

[0017] Furthermore, the above-mentioned image recognition triggering method is implemented through the following steps:

[0018] Image acquisition: Take photos of on-site components and problems using a mobile terminal camera, and simultaneously collect environmental parameters, including light intensity, shooting distance, and equipment pitch angle;

[0019] Image preprocessing: Gaussian filtering is used to remove image noise, and the edge contours of components are extracted using the Canny edge detection algorithm; the Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the mobile terminal camera, and the image distortion is corrected by the intrinsic parameter matrix to eliminate perspective error;

[0020] BIM Component-Specific Multi-Dimensional Feature Extraction: Extracts 3 types of core features and 2 types of auxiliary features for BIM components. The feature dimensions are specifically designed for the geometric characteristics of building components to avoid redundant features.

[0021] BIM Model Feature Library Pre-construction: Extract the above 5 types of features of all components in the BIM model in advance, and build a BIM component feature library according to building-floor-component type-component ID, and store it synchronously to the edge lightweight AI and the cloud core AI; the fields should at least include: component ID, geometric shape features, structural texture features, spatial location features, color features, and number of feature points.

[0022] Weighted Euclidean distance feature matching algorithm: Calculate the weighted Euclidean distance between the component features in the on-site captured image and the features in the BIM component feature library, and the one with the smallest distance is the matching result;

[0023] Matching result determination: If the matching result does not exceed the set first distance threshold, it is determined to be an exact match, and the component ID is directly located; if the matching result is between the first distance threshold and the second distance threshold, the Top 3 matching results are output and the user can manually select; if the matching result exceeds the second distance threshold, the matching fails, and image acquisition is repeated until it does not exceed the second distance threshold; the second distance threshold is greater than the first distance threshold.

[0024] Furthermore, the above three core features include geometric shape features, structural texture features, and spatial location features; the two auxiliary features include color features and feature point data, wherein:

[0025] The geometric shape features are obtained by using the RotatedRect algorithm to obtain the minimum bounding rectangle of the component, and the aspect ratio, area and perimeter of the rectangle are calculated. The seven Hu moment invariants of the contour are extracted and the absolute value logarithmic transformation of the seven Hu moment invariants is performed to eliminate the difference in order of magnitude, and they are used as geometric shape features.

[0026] The structural texture features are obtained by using the unified mode LBP operator. Based on preset parameters, the LBP texture histogram of the component region is extracted, and the 256-dimensional features are reduced to 59-dimensional unified mode features as structural texture features.

[0027] The spatial location feature calculation calculates the relative proportion of the center coordinates of the smallest bounding rectangle of the component in the image. Combined with the pitch angle and roll angle obtained by the gravity sensor of the shooting device, the initial position feature of the component is corrected to determine the relative position of the component in the floor, which is used as the spatial location feature.

[0028] Color features are achieved by converting the image from RGB to HSV color space, quantizing the hue channel at 36 levels, and quantizing the saturation and lightness channels at 5 levels each, merging them into a 90-dimensional color histogram, and taking the dominant hue corresponding to the peak of the histogram as the color feature.

[0029] The number of feature points is determined by extracting key feature points from the component region using the SIFT algorithm and counting the number of valid feature points.

[0030] Furthermore, the aforementioned weighted Euclidean distance is specifically as follows: ,in: ; In the formula, For weighted Euclidean distance, Let represent the weight of the nth class feature, and ; This represents the normalized value of the nth class feature extracted from the image. This represents the normalized value of the nth type of feature in the BIM model feature library. For feature number, These correspond to geometric shape features, structural texture features, spatial location features, color features, and the number of feature points, respectively. The weights are adjusted using a correction factor. is the correction coefficient for the nth type of feature.

[0031] Furthermore, the aforementioned distributed AI parsing layer parses the on-site data, including multi-task complex instruction decomposition, multi-terminal collaborative scheduling, and multi-modal interaction intent alignment; among which, multi-modal interaction intent alignment is achieved through the following steps:

[0032] Multimodal single-channel feature extraction: For N types of modal data input synchronously by the user, single-modal feature vectors are extracted through the corresponding encoding models, and the initial intent distribution of each modality is output;

[0033] Modal confidence weight calculation: Based on the recognition confidence and environmental parameters of each modality, calculate the confidence weight of each modality. In the formula, Let i be the recognition confidence level for the i-th modality. The environment adaptation coefficient for the i-th mode;

[0034] Global intent fusion distribution calculation: Based on credibility weights, the initial intent distributions of all modalities are weighted and fused to obtain the global intent fusion distribution. and take each The largest category is used as the global core intent;

[0035] BIM Knowledge Graph Intent Constraint Verification: Based on the global core intent, the constraint rules and entities corresponding to the intent are retrieved from the BIM knowledge graph, and the feature vector of each modality is constrained and verified, eliminating features that are irrelevant to the global core intent.

[0036] Cross-modal intent alignment feature fusion: Using a contrastive learning approach, global core intents are used as positive samples and irrelevant intents are used as negative samples. Features from all modalities are aligned and optimized to obtain an aligned cross-modal fused feature vector.

[0037] Intent parsing result output: Based on the aligned cross-modal fusion feature vector, the core slot parameters are extracted, and the final intent parsing result is output to complete the intent alignment of multimodal interaction.

[0038] Furthermore, the above-mentioned multi-source business data is integrated through a multi-source data fusion layer, and a BIM knowledge graph is constructed with component ID as the core node, specifically including:

[0039] Entity Construction: Based on the BIM model analysis results, construct the core entities of the components, connect with the construction management system, acceptance specification library, and material and equipment system, extract structured data, and construct related entities;

[0040] Entity alignment and disambiguation: Using component ID as the unique primary key, entities from different data sources are aligned, and issues of identical names and synonyms are eliminated;

[0041] Relation extraction: Using rule matching and a fine-tuned small language model, semantic relationships between entities are extracted;

[0042] Hierarchical index construction: Build a composite hierarchical index by building-floor-component type-component ID;

[0043] Knowledge graph storage: The knowledge graph is stored using a MongoDB database, and a composite index of component ID, building number, and floor number is constructed.

[0044] Furthermore, the basic interactions with the on-site model include precise component positioning, highlighting and / or box selection marking, and visual annotation of problems; model analysis and on-site collaboration are achieved through the following steps:

[0045] Prior information binding: In the BIM model, the actual size prior information is bound to each component, including the component's design length, width, height, and the actual spatial distance between component feature point pairs;

[0046] Component location: Take photos of on-site components using a mobile terminal, and identify the corresponding BIM component ID through image feature matching;

[0047] Image scale factor calculation: Extract feature point pairs in the image that correspond to prior information, calculate the pixel distance between the feature point pairs, and combine the actual spatial distance of the component to calculate the image scale factor;

[0048] Actual dimensions and spacing calculation on site: Extract the edge contour of the component to be measured from the image, calculate the pixel distance by the pixel coordinates of the two endpoints of the spacing to be measured, and obtain the actual dimensions of the component and the actual spacing between the components by multiplying the pixel distance by the scale factor.

[0049] Compliance comparison: Extract the design parameters, acceptance criteria and spatial relationship data of the component from the BIM structured database and knowledge graph, and compare them with the on-site measured data;

[0050] Output results: If the test fails, an early warning message is generated and the component is highlighted in the lightweight model, and the problem description, test results and standard references are linked simultaneously.

[0051] Secondly, this application provides a model interaction and data fusion system based on AI intelligent agents and multimodal deep learning. This system is implemented based on a multi-layer architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer. The system includes:

[0052] The on-site data acquisition module is used to acquire on-site data through various triggering methods of the multimodal input layer, including text commands, voice commands, image recognition, on-site GPS positioning, and component scanning.

[0053] The on-site data analysis module is used to analyze on-site data using a distributed AI analysis layer. The distributed AI analysis layer adopts a collaborative analysis system of lightweight AI at the edge and core AI in the cloud.

[0054] The graph construction module is used to build a BIM knowledge graph based on the parsing results of the distributed AI parsing layer and the fusion of multi-source business data through the multi-source data fusion layer, with the component ID as the core node.

[0055] The interaction module is used to perform basic interaction with the site model, model analysis and site linkage, and real-time collaborative interaction with multiple terminals through the lightweight model interaction layer based on the BIM knowledge graph.

[0056] The linkage module is used to process the interaction results based on the lightweight model interaction layer, and to carry out quality rectification closed-loop linkage, technical briefing linkage, progress control linkage, and data archiving linkage through the business scenario linkage layer.

[0057] Thirdly, this application provides an electronic device, including: at least one processor, at least one memory, and a data bus;

[0058] In this system, the processor and memory communicate with each other via a data bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the model interaction and data fusion method based on AI intelligent agents and multimodal deep learning, as described in any of the first aspects.

[0059] Fourthly, this application provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the model interaction and data fusion method based on AI intelligent agents and multimodal deep learning as described in any of the first aspects.

[0060] Compared with the prior art, the present invention has at least the following beneficial effects:

[0061] 1. Lowering the barrier to entry for BIM and significantly improving interaction efficiency: Supports six types of multimodal interaction triggering methods, requiring no professional BIM operation skills, allowing workers, inspectors, and managers to quickly get started; component data can be retrieved in seconds, and the efficiency of multimodal triggering and positioning is improved by more than 80%; the lightweight model is compatible with mobile devices such as mobile phones and tablets, and supports offline interaction in environments without or with weak network connectivity, fully adapting to the needs of on-site construction and truly promoting the widespread adoption of BIM technology.

[0062] 2. Deep integration of multi-source data to upgrade from "data retrieval" to "intelligent decision-making": By integrating multi-source data through BIM knowledge graph, not only can one-stop retrieval of basic component data and related data be achieved, but also intelligent rectification suggestions can be provided for on-site problems based on knowledge graph reasoning. This upgrades AI from an "instruction converter" to a "data decision-maker", increasing BIM data utilization by more than 95% and fully releasing the value of all BIM data.

[0063] 3. Deep integration of on-site issues and components enables integrated management of "issue-model-data": AI accurately locates components based on multimodal data and completes visual annotation. The annotation data is permanently bound to the component ID and synchronized to the structured database. It supports quick retrieval of historical issues through dimensions such as component ID, building, and floor, forming a closed loop of "annotation-storage-traceability" management, reducing human error and cumbersome processes, and improving the accuracy of issue location by 100%.

[0064] 4. Multi-dimensional model interaction, adapted to multi-person collaborative scenarios on construction sites: Supports real-time collaborative annotation and online discussion across multiple terminals, overcoming the limitations of single-terminal operation in existing solutions, and improving work efficiency by more than 70% in collaborative scenarios such as technical briefings and multi-person inspections; Advanced interactive functions enable deep linkage between the model and on-site construction, enhancing the engineering application value of BIM models.

[0065] 5. Interactive results are integrated into the business closed loop, significantly improving project management efficiency: BIM interactive results are directly converted into rectification work orders, technical briefing documents, etc., eliminating the need for manual secondary entry. This enables full-link business management from "problem labeling - rectification assignment - acceptance confirmation - data traceability," improving the efficiency of project management processes such as quality / progress control by more than 70%. Data is uniformly stored in the BIM information management platform and linked with modules such as inspection, rectification, and project management, providing accurate and comprehensive data support for enterprise management.

[0066] 6. The technical solution has strong compatibility and broad cross-domain adaptability: The lightweight engine supports the replacement of open source / commercial products such as BIMserver, and the AI ​​model supports the adaptation of various large language models. It can be realized by simply adjusting the parameter matching rules. The core technology can be extended to engineering construction fields such as municipal, highway, water conservancy, and rail transit, and can also be applied to the intelligent interaction and data management of industrial equipment BIM, park planning BIM, municipal pipeline network BIM and other models. Attached Figure Description

[0067] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0068] Figure 1 This is a flowchart of the fusion method in an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the connection of the fusion system in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the connection of an electronic device in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0072] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0073] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0074] In the description of the embodiments of the present invention, "multiple" means at least two.

[0075] Example 1: This example provides a model interaction and data fusion method based on AI agents and multimodal deep learning. This method is executed through a multi-layer architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer, as Figure 1 shown. The method includes the following specific steps:

[0076] S111. Obtain on-site data through various triggering methods of the multimodal input layer. The triggering methods include text instructions, voice instructions, image recognition, on-site GPS positioning, and component scanning.

[0077] Among them, the multimodal input layer breaks through the single text / voice instruction mode and supports six interaction triggering methods: text instruction, voice instruction, on-site image recognition, on-site GPS positioning, and component scanning triggering, covering all non-standard operation scenarios on-site. Specifically as follows:

[0078] 1. Text instruction triggering; adopt the Jieba word segmentation + custom BIM professional word library forced matching mode for instruction parsing. The custom word library has a higher priority than the general word library to avoid incorrect segmentation of engineering professional vocabulary; the custom word library contains engineering-specific vocabulary (such as pile caps, construction columns, beam bodies, etc.) and project-specific vocabulary (such as building numbers, component types, construction areas, etc.); the specific parsing process:

[0079] S1. Text instruction input: Receive the natural language text instruction input by the user;

[0080] S2. Lexical matching and word segmentation: Complete instruction word segmentation through BIM professional word library matching to avoid incorrect splitting of professional vocabulary;

[0081] S3. Remove stop words: Adopt an engineering-specific stop word list to剔除 general stop words (such as "de, le") and meaningless words in the engineering scenario (such as "that, help, take a look") to reduce invalid features;

[0082] S4. Core parameter extraction: Adopt the slot filling method to pre-define 6 types of core slots for BIM interaction: intent (location, query, detection, annotation, disclosure, rectification), component (pile cap, construction column, beam, etc.), space (building number, floor, construction area), attribute dimensions, material, elevation, acceptance specification), time (construction stage, progress date), personnel (construction team, person in charge), and extract the parameter values of the corresponding slots through BIM word library matching. Slots without values are marked as "to be supplemented".

[0083] 2. Voice instruction triggering; adopt voice recognition API + BIM voice instruction noise reduction processing, optimized for the high-noise environment on the construction site. The specific implementation process:

[0084] S1. Voice stream acquisition: The user's voice commands are acquired through the mobile terminal's microphone, and environmental noise samples are acquired simultaneously.

[0085] S2. Noise Reduction Preprocessing: The speech stream is denoised using spectral subtraction. First, the speech signal is converted into a frequency domain signal through Fast Fourier Transform. Then, the estimated noise power spectrum is subtracted from the noisy speech power spectrum. Finally, the clean speech signal is restored through Inverse Fourier Transform to solve the problems of construction site machinery noise and environmental noise interference.

[0086] S3, Speech-to-Text: Converts noise-reduced clean speech streams into text content using a speech recognition API;

[0087] S4. Command Parsing: The subsequent process is completely consistent with the parsing process triggered by the above text command, and the core parameters are extracted.

[0088] 3. On-site image recognition trigger; a multi-dimensional feature extraction + weighted Euclidean distance feature matching algorithm is specifically designed for the geometric features of BIM components, solving the problems of poor adaptability and low positioning accuracy of general image matching algorithms for BIM components. Specific implementation process:

[0089] S1. Image Acquisition: Take photos of on-site components / problems using the mobile terminal's camera, and simultaneously collect environmental parameters (light intensity, shooting distance, equipment tilt angle).

[0090] S2. Image preprocessing: Gaussian filtering is used to remove image noise, and the edge contours of components are extracted using the Canny edge detection algorithm to eliminate background interference. At the same time, Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the mobile terminal camera, and the image is distorted by the intrinsic parameter matrix to eliminate perspective error.

[0091] S3. BIM Component-Specific Multi-Dimensional Feature Extraction: Extracts 3 types of core features + 2 types of auxiliary features from BIM components. The feature dimensions are specifically designed for the geometric characteristics of building components to avoid redundant features. The specific extraction rules are as follows:

[0092] Core Feature 1 (Geometric Shape Features): The minimum bounding rectangle of the component is obtained using the RotatedRect algorithm, and the aspect ratio, area, and perimeter of the rectangle are calculated. Simultaneously, seven Hu moment invariants of the contour are extracted to ensure feature invariance after rotation / scaling. The core formula is:

[0093] Q1=η20+η02; Q2=(η20-η02)2+4η112; Q3=(η30-3η12)2+(3η21-η03)2;

[0094] Where η is the normalized central moment, and finally the seven Hu moments are transformed by the absolute value logarithm to eliminate the difference in magnitude, and used as the geometric shape feature vector;

[0095] Core Feature 2 (Structural Texture Feature): Using the unified mode LBP operator, with radius R=1 and neighborhood point number P=8, the LBP texture histogram of the component region is extracted, reducing the 256-dimensional feature to a 59-dimensional unified mode feature as the structural texture feature vector, distinguishing components of different materials such as concrete, steel structure, and masonry.

[0096] Core Feature 3 (Spatial Position Feature): Calculate the relative proportion of the center coordinates of the minimum bounding rectangle of the component in the image (x_center / image width, y_center / image height), and combine it with the pitch angle and roll angle obtained by the gravity sensor of the shooting device to correct the spatial position feature of the component and make a preliminary judgment on the relative position of the component in the floor.

[0097] Auxiliary Feature 1 (Color Feature): Convert the image from RGB to HSV color space, perform 36 levels of quantization on the H (hue) channel, and perform 5 levels of quantization on the S (saturation) and V (brightness) channels respectively, and merge them into a 90-dimensional color histogram. Take the dominant hue corresponding to the peak of the histogram as the color feature.

[0098] Auxiliary Feature 2 (Number of Feature Points): The SIFT algorithm is used to extract key feature points in the component area and count the number of effective feature points as an auxiliary feature to distinguish between components with dense feature points such as steel bars and formwork and components with sparse feature points such as concrete walls.

[0099] S4. Pre-construction of BIM Model Feature Library: Extract the above 5 types of features from all components in the BIM model in advance, and construct a BIM component feature library according to building-floor-component type-component ID. This library is synchronously stored on the edge and cloud. The feature library data format is JSON, containing fields such as: component ID, geometric shape feature, structural texture feature, spatial location feature, color feature, and number of feature points. All feature values ​​are normalized to the [0,1] interval using min-max to eliminate differences in the dimensions of different features. The normalization formula is:

[0100] Xnorm = (Xmax - Xmin) / (X - Xmin), where Xmin and Xmax are the minimum and maximum values ​​of this type of feature in the entire project feature library;

[0101] S5. Weighted Euclidean Distance Feature Matching Algorithm: Calculates the weighted Euclidean distance between the component features in the on-site captured image and the features in the BIM component feature library. The one with the smallest distance is the matching result. By default, the core feature weight accounts for 80%, and the auxiliary feature weight accounts for 20%. The complete matching formula is: ; In the formula, For weighted Euclidean distance, Let represent the weight of the nth class feature, and ; This represents the normalized value of the nth class feature extracted from the image. This represents the normalized value of the nth type of feature in the BIM model feature library. For feature number, These correspond to geometric shape features, structural texture features, spatial location features, color features, and the number of feature points, respectively.

[0102] Among them, weight It is not a static, fixed value, but rather dynamically and adaptively adjusted based on environmental parameters such as ambient lighting, shooting distance, and component obstruction. Specifically, it is implemented as follows:

[0103] The weight adjustment is achieved by analyzing the input image in real time using a pre-trained lightweight environment perception model, and then outputting a weight adjustment vector. The model uses a 4-layer lightweight CNN classifier with the following structure: Input → Conv1 (3×3 convolution, 16 convolution kernels, stride 2, ReLU) → MaxPool (2×2) → Conv2 (3×3 convolution, 32 convolution kernels, stride 2, ReLU) → MaxPool (2×2) → Conv3 (3×3 convolution, 64 convolution kernels, stride 2, ReLU) → GlobalAvgPool → FC1 (64-dimensional, ReLU) → FC2 (5-dimensional, Sigmoid) → Output. The output is a 5-dimensional weight correction coefficient k1~k5, corresponding to the weight correction coefficients of the 5 feature classes, with a range of [0.5, 1.5].

[0104] Model training rules: Collect 10,000 images of components in different environments at the construction site, label them with environmental tags (light intensity, shooting distance, degree of occlusion) and corresponding optimal feature weights, and perform data augmentation through random flipping, brightness adjustment, noise addition, and occlusion simulation to expand to 50,000 images; use the Adam optimizer with an initial learning rate of 1e-4, a batch size of 32, a loss function of mean squared error (MSE), 50 training epochs, and an early stopping strategy to prevent overfitting;

[0105] The model outputs 5-dimensional weight correction coefficients k1-k5, corresponding to the original weights w1-w5 of the 5 feature classes. The corrected weights are: ; Ensure that the sum of the corrected weights is always 1, satisfying: when the ambient light is lower than 50 lux, k4 (color feature correction coefficient) automatically drops to 0.5; when the shooting distance is greater than 3 m, the sub-feature correction coefficients corresponding to the area and perimeter in the geometric shape features drop to 0.6, and the sub-feature correction coefficients corresponding to the aspect ratio and Hu moments rise to 1.4; when the component occlusion rate > 30%, the correction coefficient of k1 (geometric shape feature) drops to 0.7, and the correction coefficient of k2 (texture feature) rises to 1.3;

[0106] S6. Matching result determination: Set the distance threshold D0 = 0.15. If D ≤ D0, it is determined as an accurate match and directly locate to the component ID; if 0.15 < D ≤ 0.3, output the top 3 matching results for the user to select manually; if D > 0.3, prompt that the matching fails and suggest changing the shooting angle.

[0107] 4. On-site GPS positioning trigger; Through the alignment of the reference of the WGS84 geodetic coordinate system and the BIM project coordinate system, achieve the accurate matching of on-site positioning and the spatial unit of the BIM model. The specific implementation process:

[0108] S1. Coordinate system reference alignment: Collect the two sets of coordinates of 3 or more identical control points at the project site in advance, which are the engineering coordinate system coordinates (X, Y, H) adopted by the BIM model and the spatial rectangular coordinates (X, Y, Z) obtained by converting longitude and latitude;

[0109] S2. BIM model spatial partitioning: Divide the BIM model into the smallest spatial units according to building - floor - construction partition, and each unit is bound with a unique partition code and the central engineering coordinate system coordinates of the unit;

[0110] S3. GPS positioning collection: Obtain the on-site GPS real-time coordinates through a mobile terminal,

[0111] S4. Accuracy verification: Set the coordinate conversion error threshold ≤ 5 cm. If the error after conversion exceeds the threshold, prompt for repositioning;

[0112] S5. Spatial matching: Calculate the spatial Euclidean distance between the calibrated on-site coordinates and the central coordinates of each BIM model spatial unit. The unit with the smallest distance is the target area, retrieve all components in this area and display them in the model.

[0113] 5. Component barcode scanning trigger, including the following steps:

[0114] S1. QR code generation: Use the草料QR code tool to generate a unique QR code for each BIM component. The core information of the QR code is encrypted and stored: project code, building number, floor number, component type, component ID. Set the QR code scanning permission verification, and only authorized project personnel can parse it;

[0115] S2. Scan and parse: The mobile terminal scans the QR code, completes the permission verification, parses the QR code content, and extracts the core component ID;

[0116] S3. Data Retrieval and Model Positioning: Based on the component ID, directly retrieve the full data of the component in the BIM model, and simultaneously drive the lightweight model to highlight and position the component, displaying all component attribute information.

[0117] S112 utilizes a distributed AI analysis layer to analyze on-site data. The distributed AI analysis layer adopts a collaborative analysis system of lightweight AI at the edge and core AI in the cloud.

[0118] The distributed AI parsing layer employs a collaborative parsing system combining lightweight edge AI and core cloud AI to address issues such as unstable construction site networks and low parsing efficiency across multiple terminals. It also features a unique multimodal interaction intent alignment method based on BIM knowledge graph enhancement, resolving pain points like inconsistent cross-modal intents and low parsing accuracy. The specific implementation is as follows:

[0119] 1. Cloud-edge AI deployment architecture;

[0120] Edge: Deploy lightweight AI models compressed using FP8 quantization technology (such as Qwen3.5-0.8B), adapted to mobile terminals such as smartphones and tablets, to achieve local lightweight parsing, including simple instruction word segmentation, image feature matching, basic operations of local lightweight models, and local graph inference. Core interactions can be completed independently in a network-free environment.

[0121] Cloud-based: Deploy an AI agent with a large language model at its core, using Qwen3.5-9B as the basic model, and fine-tuning it based on BIM professional corpus. It supports concurrent requests from multiple terminals and achieves complex parsing and global collaboration, including multi-task complex instruction decomposition, multi-modal data fusion parsing, multi-terminal collaborative scheduling, knowledge graph reasoning, and complex business instruction conversion.

[0122] 2. Core parsing function implementation: This includes the process of breaking down complex multi-task instructions:

[0123] S1, Instruction Intent Recognition: Input the natural language instructions input by the user into the fine-tuned intent recognition model, and output the core intent and task dependencies;

[0124] S2. Task Dependency Graph Construction: Based on the dependency relationships between tasks (such as "first locate the component → then conduct compliance inspection → finally generate the rectification work order"), construct a directed acyclic graph (DAG) to clarify the task execution order;

[0125] S3, Cloud-Edge Task Distribution: Based on the cloud-edge task division criteria, subtasks in the DAG are distributed to the edge or cloud for execution.

[0126] S4. Task Execution and Status Monitoring: Real-time monitoring of subtask execution status. If a subtask fails to execute, it will automatically retry twice. If it still fails, it will be marked as an exception and pushed to the user for manual intervention.

[0127] S5. Execution Result Aggregation: After all subtasks are completed, the execution results from the edge and the cloud are aggregated to output a unified interactive result.

[0128] Among them, multi-terminal collaborative scheduling:

[0129] A collaborative scheduling mechanism is built using WebSocket long connections, with the cloud serving as the scheduling center to receive operation instructions from each terminal and broadcast operation information synchronously. An optimistic locking mechanism is used to resolve multi-terminal operation conflicts. For the same component ID, only one operation instruction is processed at a time, and they are executed in the order of timestamps to ensure that the state of the multi-terminal model is consistent.

[0130] Among them, complex business instruction conversion:

[0131] By using a three-step approach of "intent recognition + scenario matching + parameter extraction", we can identify business scenarios such as quality inspection, technical briefing, and progress control, extract core parameters, and convert natural language commands into standardized commands that can be recognized by the business system.

[0132] Cloud-edge collaboration rules: When online, the edge device uploads complex data to the cloud, and the cloud parses the data and returns the results, which are then executed by the edge device; when offline, the edge device interacts locally, stores operation logs locally, and automatically synchronizes the data to the cloud after connecting to the network to ensure data consistency.

[0133] 3. A multimodal interaction intent alignment method based on BIM knowledge graph enhancement addresses the pain points of inconsistent cross-modal intents and low parsing accuracy in existing multimodal BIM interactions. The specific steps are as follows:

[0134] S1. Multimodal Single-Channel Feature Extraction: For N types of modal data (text, voice, image, GPS, QR code) input by the user simultaneously, extract single-modal feature vectors F1, F2, ..., FN through the corresponding encoding model, and output the initial intent distribution P1, P2, ..., PN for each modality, where Pi = [pi1, pi2, ..., piM], and M is the number of predefined BIM interaction intent categories (location, query, detection, annotation, briefing, rectification).

[0135] S2. Modal confidence weight calculation: Based on the recognition confidence and environmental parameters of each modality, calculate the confidence weight of each modality. The formula is: In the formula, Let be the recognition confidence score for the i-th modality (e.g., confidence score for speech recognition, confidence score for image matching). Let be the environment adaptation coefficient for the i-th modality, ranging from [0,1], representing the image modality under low illumination conditions. The value dropped to 0.6, indicating a decrease in speech modality in high-noise environments. It dropped to 0.7;

[0136] S3. Global Intent Fusion Distribution Calculation: Based on the credibility weights, the initial intent distributions of all modalities are weighted and fused to obtain the global intent fusion distribution Pglobal, as shown in the formula: ; Take each The largest category is taken as the global core intent, which is represented as Iglobal;

[0137] S4. BIM Knowledge Graph Intent Constraint Verification: Based on the global core intent Iglobal, the constraint rules and entities corresponding to the intent are retrieved from the BIM knowledge graph. The feature vector of each modality is constrained and verified, and features that are irrelevant to the global core intent are removed.

[0138] S5. Cross-modal intent alignment feature fusion: Using a contrastive learning approach, global core intents are used as positive samples and irrelevant intents are used as negative samples. Features of all modalities are aligned and optimized to obtain the aligned cross-modal fusion feature vector Falign.

[0139] S6. Intent parsing result output: Based on the aligned fused feature vector, extract the core slot parameters, output the final intent parsing result, and complete the intent alignment for multimodal interaction.

[0140] S113, based on the parsing results of the distributed AI parsing layer, integrates multi-source business data through the multi-source data fusion layer, and constructs a BIM knowledge graph with component ID as the core node.

[0141] The multi-source data fusion layer parses original BIM models such as RVT / IFC, extracts core component data, integrates multi-source business data, and constructs a BIM knowledge graph with component ID as the core node, realizing an upgrade from "data retrieval" to "intelligent decision-making." The specific implementation is as follows:

[0142] (1) BIM model analysis and lightweight processing:

[0143] Using open-source projects such as the BIMserver lightweight engine, the original RVT / IFC BIM model is parsed, and component geometric data, attribute data, and spatial topology association data are extracted. The model is lightweighted through triangular facet simplification and instantiation reuse technology, generating a lightweight model adapted to web and mobile terminals. The model is stored in a five-level hierarchical manner: "project level - building level - floor level - zone level - component level".

[0144] (2) Construction of BIM knowledge graph:

[0145] By integrating multi-source data such as BIM model data, on-site construction progress, engineering quality acceptance specifications, component historical maintenance / issues, materials and equipment, and personnel teams, a BIM knowledge graph is constructed with component IDs as the core nodes. The specific implementation is as follows:

[0146] Core Entity: Component Entity (Core Node, Attributes: Component ID, Project Code, Building Number, Floor Number, Component Type, Geometric Parameters, Material, Design Elevation, Construction Team, Arrival Time, Acceptance Status);

[0147] First-level related entities: construction data entity, acceptance specification entity, progress plan entity, problem rectification entity, and materials and equipment entity;

[0148] Second-level related entities: Personnel entity, Work team entity, Machinery and equipment entity, Document entity;

[0149] Core semantic relationships: Component → Must meet → Acceptance specifications; Component → Work team; Component → Problem rectification form; Component → Schedule plan; Work team → Construction unit;

[0150] The above knowledge graph construction process:

[0151] S1. Entity Construction: Based on the BIM model analysis results, construct the core entities of the components, connect with the construction management system, acceptance specification library, and material and equipment system, extract structured data, and construct related entities;

[0152] S2. Entity alignment and disambiguation: Using the component ID as the unique primary key, entities from different data sources are aligned to eliminate the problems of homonyms and synonyms.

[0153] S3. Relation Extraction: Using rule matching and a fine-tuned small language model, semantic relationships between entities are extracted.

[0154] S4. Hierarchical Index Construction: Create a composite hierarchical index based on building-floor-component type-component ID;

[0155] S5. Knowledge Graph Storage: The knowledge graph is stored using a MongoDB database, and a composite index of component ID, building number, and floor number is constructed to improve retrieval efficiency.

[0156] (3) Intelligent decision-making and reasoning based on knowledge graphs

[0157] Using component ID as the core node, the system traverses the associated nodes of the BIM knowledge graph to achieve intelligent decision-making. The specific process is as follows:

[0158] S1. Input: Description of the on-site problem, component ID, and on-site measured data;

[0159] S2, Subgraph Retrieval: Traverse the knowledge graph using component ID as the core node;

[0160] S3. Execution Decision: Load predefined reasoning rules, as shown in the following examples: Rule 1: Measured spacing of components < minimum spacing requirement in the specification → Determine as unqualified → Trigger rectification suggestion: Adjust the component spacing to the range required by the specification, and simultaneously issue rectification basis; Rule 2: Deviation between measured dimensions and design dimensions of components > 5% → Determine warning → Trigger review suggestion: Review the processing and installation accuracy of components;

[0161] S4. Output Results: Synchronize with the original specification text to ensure traceability.

[0162] S114, based on the BIM knowledge graph, enables basic interaction with the site model, model analysis and site linkage, and real-time collaborative interaction across multiple terminals through a lightweight model interaction layer.

[0163] The lightweight model interaction layer, based on a lightweight engine (supporting replacement of open-source / commercial engines such as BIMserver, XBIM, and FZKViewer), drives the lightweight model to complete multi-dimensional interactions including basic operations, advanced operations, and collaborative operations. The specific implementation is as follows:

[0164] (1) Basic operation: Basic interaction with the model in conjunction with the on-site environment;

[0165] Precise component positioning: Based on the component ID queried by AI, the lightweight model engine search and positioning function is used to adjust the model view to the location of the component, adapt to the best observation angle, and directly correspond to the location of the component on site.

[0166] Highlight / Selection Marker: Highlight and select target components, and the marker color can be customized. It is synchronously associated with the components marked on site to achieve a precise correspondence between the model and the components on site.

[0167] Problem visualization annotation: A 3D annotation plugin based on Three.js is used to generate 3D annotation icons at the corresponding component positions on the model. The annotation information (problem description, annotator, annotation time, rectification requirements) is permanently bound to the component ID. The icons change synchronously when the model is scaled or rotated, realizing real-time linkage between on-site problems and model annotations, and the annotation information is synchronously associated with on-site problem data.

[0168] (2) Advanced Operations: Intelligent Model Analysis and On-site Interaction;

[0169] Component spacing compliance inspection: Achieving a closed-loop process from on-site image acquisition to compliance determination, the specific steps are as follows:

[0170] S1. Prior information binding: In advance, bind the actual size prior information of each component in the BIM model, including the component's design length L0, width W0, height H0, and the actual spatial distance D0 between component feature point pairs;

[0171] S2. Component Location: Users take photos of on-site components using mobile terminals, and the corresponding BIM component IDs are identified through image feature matching.

[0172] S3. Image scale factor calculation: Extract feature point pairs in the image that correspond to prior information, calculate the pixel distance dpixel of the feature point pairs, and combine it with the actual spatial distance D0 of the component to calculate the image scale factor s: s=D0 / dpixel (unit: mm / pixel).

[0173] S4. Calculation of actual dimensions and spacing on site: Extract the edge contour of the component to be measured and the pixel coordinates of the two endpoints of the spacing to be measured from the image, calculate the pixel distance, multiply by the scale factor s, and obtain the actual dimensions of the component on site and the actual spacing between the components.

[0174] S5. Compliance Comparison: Extract the design parameters, acceptance criteria, and spatial relationship data of the component from the BIM structured database and knowledge graph, and compare them with the on-site measured data;

[0175] S6. Result Output: If the test fails, an early warning message will be automatically generated and the component will be highlighted in the lightweight model. The problem description, test results and standard references will be linked simultaneously.

[0176] Real-time linkage between on-site and model perspectives: Using AR visual positioning algorithms, the perspective of the mobile terminal camera is calibrated in real time with the perspective of the BIM model, and the information of BIM model components is superimposed to achieve synchronous correspondence between the on-site perspective and the model perspective, thus completing AR technology briefing and comparison between the on-site and model.

[0177] Phased display of construction progress model: Integrating with the progress management system, the lightweight model is driven to display completed, under construction, and unstarted components according to the construction progress plan and actual completion status. Components at different stages are marked with different colors to achieve visual management of progress.

[0178] (3) Collaborative operation: Real-time collaborative interaction across multiple terminals

[0179] It supports real-time collaborative annotation and online discussion of the same model by multiple terminal users, and the cloud uniformly schedules the status of all terminal models, records the operator, operation time and operation content, and realizes full-link traceability of operation; the interaction results are synchronized to the BIM structured database in real time, realizing deep linkage between data and model.

[0180] S115, based on the interaction processing results of the lightweight model interaction layer, enables closed-loop linkage for quality rectification, technical briefing, progress control, and data archiving through the business scenario linkage layer.

[0181] The business scenario linkage layer uses interface adaptation technology to directly connect the BIM model interaction / problem annotation results to the engineering management business system (smart construction site platform, quality / progress control system, project management system), eliminating the need for manual secondary data entry and forming a closed business loop. The specific implementation is as follows:

[0182] Closed-loop linkage for quality rectification: After a problem is marked, a rectification work order is automatically generated, which includes component ID, problem description, AI rectification suggestions, acceptance standard basis, and rectification time limit. It directly connects to the project quality control system through RESTful API, dispatches the work order to the corresponding construction team, and synchronizes the rectification and acceptance status of the work order to the BIM platform in real time, completing the entire closed loop of "marking-dispatch-rectification-acceptance-archiving".

[0183] Linked technical briefing: The component data query results can be exported as a standardized technical briefing document with one click, which includes the full design parameters of the component, construction requirements, acceptance specifications, and information on the association between adjacent components, without the need for manual organization of BIM data;

[0184] Progress control linkage: Progress-related interaction results are synchronized to the progress management system, enabling comparative analysis of BIM model progress and actual construction progress. When the progress deviation exceeds the preset threshold, a deviation warning is automatically generated and pushed to project management personnel.

[0185] Data archiving and linkage: All interaction results, problem rectification data, and business documents are permanently bound to the component ID and synchronously stored in the structured database of the BIM information management platform, supporting full lifecycle traceability and providing data support for project settlement, operation and maintenance management, and enterprise data analysis.

[0186] Optionally, the above method may also include an online / offline dual-mode core process:

[0187] The dual-mode inference mechanism of "real-time prediction on the edge + precise verification on the cloud" is adapted to construction site scenarios with no or weak network coverage, ensuring business continuity. The specific process is as follows:

[0188] (1) Core processes in online mode;

[0189] Multimodal triggering → initial parsing at the edge → complex parsing / knowledge graph reasoning in the cloud → multi-dimensional interaction of lightweight models → closed-loop linkage of business systems → multi-terminal collaborative synchronization.

[0190] (2) Core process of offline mode;

[0191] Multimodal triggering → local parsing / basic interaction at the edge → local data storage → cloud-edge data synchronization after network connection → cloud-based parsing / knowledge graph reasoning → business system integration.

[0192] (3) Core supporting technologies for offline mode;

[0193] Map construction:

[0194] S1. Area Division: Divide the area into the smallest units according to the user's commonly used construction zones (such as the current floor or the current work surface);

[0195] S2. Core Entity Filtering: Using the component ID within the region as the core, filter the core related entities within the region and eliminate unnecessary entities at the project level.

[0196] S3, Lightweight Storage: Converts graph rules into JSON format and stores them on edge mobile terminals, adapting to the storage capabilities of mobile terminals;

[0197] S4. Automatic Update: When a user switches a construction area or the cloud knowledge graph rules are updated, the graph of the corresponding area will be automatically updated when connected to the network.

[0198] Dual-mode inference process: "Edge-side prediction + cloud-side verification"

[0199] S1. In offline mode, the edge device receives on-site measured data and component ID, and loads the map and simplified rules of the corresponding area.

[0200] S2. End-side prediction: Calculate the deviation rate between measured data and specification requirements (measured value vs. design value).

[0201] If δ≤2%: it is considered qualified, the result is output directly and stored locally;

[0202] If 2% < δ ≤ 5%: a warning is issued, a preliminary conclusion is output, and it is marked for cloud verification.

[0203] If δ>5%: it is deemed unqualified, and the submission of acceptance application is prohibited. The evidence of violation will be output and marked for cloud review.

[0204] S3. After connecting to the network, it will automatically synchronize all operation data and prediction results in the offline state to the cloud.

[0205] S4, Cloud-based review: Load the full knowledge graph, execute complete rule reasoning, and combine global data (data of other components in the same layer, historical acceptance data) for accurate review;

[0206] S5. Result Correction: If the cloud review result is consistent with the terminal prediction, the business system will be updated synchronously; if they are inconsistent, the corrected result will be pushed to the user, explaining the basis for the review, and completing the data loop.

[0207] Data consistency guarantee mechanism: In offline mode, the edge uses a local SQLite database to store operation logs. Each operation log contains a unique operation UUID, operator ID, timestamp, component ID, operation type, operation content, and synchronization status. After connecting to the network, data synchronization is completed through an idempotent mechanism based on the operation UUID to avoid duplicate execution. At the same time, a conflict detection mechanism is used to resolve the conflict between cloud and local operations, ensuring data consistency between the cloud and the edge.

[0208] Optionally, the above may also include manual review and secondary interaction:

[0209] It supports manual review of interaction results, search data, and issue annotations. Secondary multimodal commands can be used to supplement queries, modify annotations, adjust model perspective / zoom, and improve work order information, further enhancing the accuracy of interaction.

[0210] Specifically, this invention adopts a technical solution of multimodal triggering + cloud-edge distributed AI collaboration + BIM knowledge graph fusion + lightweight model multi-dimensional interaction + business scenario closed-loop linkage to realize intelligent querying of BIM data and deep multimodal interaction.

[0211] The method of the present invention will be further illustrated by the following examples:

[0212] 1. In a network-free environment, images trigger a closed loop of local BIM component data query, compliance inspection, and online business processing. Application scenario: At the construction site of Building 5, 3rd floor, where there is no network, inspectors discover honeycomb-like defects in the beams. Simultaneously, they need to verify whether the beam spacing meets the specifications. This requires quickly querying the beam's foundation data and acceptance standards to complete a local compliance assessment. Subsequent online processing then traces the issue and assigns rectification orders. The specific implementation steps are as follows:

[0213] The inspection personnel opened the BIM information management platform on their mobile phones. The edge device had already downloaded the offline lightweight model of the 5 buildings with 3 floors, the FP8 quantized lightweight AI model, the component feature library and the map of the corresponding area in advance to meet the offline use requirements.

[0214] The inspection personnel took photos of the beam structure on-site and selected the image-triggered interactive mode.

[0215] Lightweight AI at the edge performs Gaussian filtering for noise reduction, Canny edge detection, and distortion correction preprocessing on the photo, extracts five types of feature vectors for the beam, performs weighted Euclidean distance matching with the local lightweight component feature library, and dynamically adjusts the feature weights. The matching result distance D=0.08≤0.15, accurately locating beam L025 of the 3rd floor of Building 5.

[0216] The edge retrieves the core data of beam L025 from the local structured database: cross-sectional dimensions 300*600mm, concrete strength C30, design beam spacing 2000mm, construction team = concrete team B, acceptance standard = "Code for Acceptance of Construction Quality of Concrete Structures" GB50204-2015, and simultaneously drives the lightweight model to highlight the beam in red.

[0217] The inspection personnel need to verify the beam spacing. The actual beam spacing of 1920mm is obtained by solving the scale factor of the monocular image at the edge end. The IF-THEN rule of the local map is loaded, and the deviation rate δ=4% is calculated. It is in the range of 2%-5%, which is judged as a warning. It is marked to be verified in the cloud and the preliminary conclusion and local prediction basis are output.

[0218] The inspection personnel marked the beam with local issues on the platform, inputting "the beam surface has honeycomb pitting, covering an area of ​​about 0.3%". The marked data, measured spacing data, and prediction results are all stored in the local SQLite database at the edge, generating a unique operation UUID and marking it as unsynchronized.

[0219] After the inspection personnel arrive at the area with network coverage, the platform automatically triggers cloud-edge data synchronization, uploading photos, component locations, problem annotations, actual measurement data, and operation logs to the cloud, and avoiding duplicate execution through UUID idempotency verification;

[0220] The cloud-based core AI performs intelligent decision-making and reasoning based on the full BIM knowledge graph, verifies the beam spacing data, and combines it with the installation data of other beams on the same floor. Finally, it determines that the beam spacing deviation is within the allowable range of the specification, maintains the warning conclusion, and at the same time matches the acceptance specification to give the rectification suggestion for honeycomb pitting: "Honeycomb pitting area < 0.5%, use 1:2 cement mortar for repair".

[0221] The cloud uses interface adaptation technology to synchronize the labeled data, review results, and rectification suggestions to the project quality control system, automatically generate rectification work orders, and assign them to concrete team B. The work order includes component ID: L025, problem description, rectification suggestions, acceptance specifications, and rectification deadline.

[0222] After rectification is completed, the work team uploads rectification photos. Once the inspection personnel approve the rectification, the work order status is updated to "completed". All data is bound to component ID: L025 and synchronized to the BIM structured database, realizing full lifecycle traceability of historical issues.

[0223] 2. In online mode, dual-modal triggering of issue labeling and intelligent order dispatching via voice and GPS positioning:

[0224] Application scenario: On the north side of the second floor of Building 3 in the project, the inspection personnel found a vertical crack in the column. Through voice + GPS positioning dual-modal trigger interaction, the entire process of multimodal intent alignment, precise component positioning, AI intelligent decision-making and rectification order dispatch was completed.

[0225] The inspectors opened the mobile platform and activated the dual-modal interaction mode of GPS positioning and voice command. The platform automatically obtained the on-site GPS spatial coordinates, and the inspectors simultaneously issued a voice command: "There is a vertical crack on the north side column of the second floor of Building 3, with a length of 1.2 meters and a width of 0.5 mm. Add a mark and note that it needs to be repaired in time."

[0226] Lightweight AI at the edge first completes the initial analysis: converting GPS coordinates into BIM engineering coordinates through seven parameters, initially matching them with the spatial coordinates of the BIM model, and locating the area on the north side of the second floor of Building 3; at the same time, spectral subtraction is used to reduce noise in the voice commands, convert them into text, complete the initial word segmentation and intent recognition, and simultaneously upload the voice data, GPS data, and initial analysis results to the cloud.

[0227] The cloud-based core AI executes a multimodal interaction intent alignment method enhanced by BIM knowledge graph. It fuses and analyzes voice and GPS dual-modal data to identify the global core intent as "quality problem annotation". The business scenario is quality inspection. The core parameters are extracted: component type = column, location = north side of the 2nd floor of building 3, problem = vertical crack 1.2m 0.5mm. Based on the BIM knowledge graph, the corresponding component ID = Z018 and the full amount of basic data (section 500500mm, concrete strength C40, construction team = rebar team C) are retrieved.

[0228] The cloud-based AI performs SWRL rule reasoning based on the knowledge graph, matches it with the "Code for Acceptance of Construction Quality of Concrete Structures" GB50204-2015, and gives intelligent rectification suggestions: "The crack width is 0.5mm. Epoxy resin grouting should be used for treatment and sealing and curing. The curing period should not be less than 7 days." At the same time, a labeling instruction data package (including component ID, labeler, time, problem description, and rectification suggestions) is generated.

[0229] The cloud sends component data, rectification suggestions, and annotation instructions back to the edge. The edge drives the lightweight model to highlight column Z018 in yellow and add a red exclamation mark 3D annotation icon. Clicking the icon allows you to view the full information, including the problem description, rectification suggestions, and standard basis.

[0230] The cloud uses interface adaptation technology to synchronize data to the project quality control system, automatically generate rectification work orders and assign them to steel reinforcement team C. The rectification status of the work orders is synchronized to the BIM platform in real time, and managers can view the rectification progress in real time.

[0231] The labeled data, rectification work order information and component ID=Z018 are permanently bound and synchronized to the BIM structured database. Managers can quickly retrieve the problem information, rectification records and acceptance results of the column by component ID, building number and floor number, so as to achieve full-chain traceability.

[0232] 3. Technical briefing on multimodal triggering of multi-component association query + multi-terminal collaboration

[0233] Application scenario: Before the electromechanical installation work on the 5th floor of Building 1, the technical supervisor needs to organize the construction team for technical briefing. By triggering multi-component association queries through text commands and combining multi-terminal collaborative functions, the online visual technical briefing can be completed.

[0234] The technical manager entered the following text command on the web-based BIM platform: "View the quantity, installation elevation, material parameters, and related wall and floor opening information of all water supply and drainage pipes on the 5th floor of Building 1".

[0235] The cloud-based AI interprets the command and identifies the core intent as "multi-component association query". The business scenario is technical briefing. It extracts the core slot parameters and retrieves the corresponding results based on the BIM knowledge graph: 32 water supply and drainage pipes in 5 floors of Building 1, including the design and installation elevation, material parameters, and pipe diameter of each pipe, as well as the location and size information of 24 reserved openings in the walls and floors.

[0236] The cloud generates model operation instructions, which drive the lightweight web model to automatically locate the 5th floor of Building 1, switch to the best overhead view, highlight all water supply and drainage pipes in blue, and mark the associated reserved openings with yellow boxes. At the same time, the interface displays a list of parameters for all pipes and openings.

[0237] The technical lead initiates a multi-terminal collaborative meeting, and construction team members join the meeting via mobile phones and tablets. The cloud uses a WebSocket long connection as the dispatch center to broadcast the technical lead's model operation instructions synchronously. The view, marked content, and displayed content of all terminal models are synchronized in real time to ensure that the status of the multi-terminal models is completely consistent.

[0238] The technical lead used a second voice command: "View the installation and acceptance specifications, construction team, and historical problem records for pipeline number G012." The cloud quickly retrieved the BIM knowledge graph, retrieved the acceptance specifications, responsible team, and historical problem rectification records for similar pipelines, and the model automatically located and highlighted pipeline number G012. The relevant information was simultaneously displayed on the interface, completing the targeted technical briefing.

[0239] After the technical briefing is completed, the technical briefing document can be exported with one click. It includes all pipeline design parameters, installation requirements, acceptance specifications, and reserved opening information. There is no need to manually organize BIM data. It can be directly distributed to the construction team. The briefing record is synchronously archived in the project management system and permanently bound to the corresponding component ID.

[0240] Example 2: This application provides a model interaction and data fusion system based on AI intelligent agents and multimodal deep learning. The system is implemented based on a multi-layer architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer, such as... Figure 2 As shown, the system includes:

[0241] The on-site data acquisition module is used to acquire on-site data through various triggering methods of the multimodal input layer, including text commands, voice commands, image recognition, on-site GPS positioning, and component scanning.

[0242] The on-site data analysis module is used to analyze on-site data using a distributed AI analysis layer. The distributed AI analysis layer adopts a collaborative analysis system of lightweight AI at the edge and core AI in the cloud.

[0243] The graph construction module is used to build a BIM knowledge graph based on the parsing results of the distributed AI parsing layer and the fusion of multi-source business data through the multi-source data fusion layer, with the component ID as the core node.

[0244] The interaction module is used to perform basic interaction with the site model, model analysis and site linkage, and real-time collaborative interaction with multiple terminals through the lightweight model interaction layer based on the BIM knowledge graph.

[0245] The linkage module is used to process the interaction results based on the lightweight model interaction layer, and to carry out quality rectification closed-loop linkage, technical briefing linkage, progress control linkage, and data archiving linkage through the business scenario linkage layer.

[0246] Example 3: This application provides an electronic device, such as... Figure 3 As shown, it includes: at least one processor, at least one memory, and a data bus;

[0247] In this system, the processor and memory communicate with each other via a data bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the model interaction and data fusion method based on AI intelligent agents and multimodal deep learning as described in Example 1.

[0248] Example 4: This application provides a non-transitory computer-readable storage medium that stores computer instructions. The computer instructions enable the computer to execute the model interaction and data fusion method based on AI intelligent agents and multimodal deep learning described in Example 1.

[0249] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0250] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0251] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0252] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0253] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0254] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for model interaction and data fusion based on AI intelligent agents and multimodal deep learning, characterized in that, This method is implemented through a multi-layered architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer. The method includes the following specific steps: Field data is acquired through various triggering methods of the multimodal input layer, including text commands, voice commands, image recognition, on-site GPS positioning, and component scanning. The distributed AI parsing layer is used to parse the on-site data. The distributed AI parsing layer adopts a collaborative parsing system of lightweight AI at the edge and core AI in the cloud. Based on the parsing results of the distributed AI parsing layer, the multi-source business data is fused through the multi-source data fusion layer, and a BIM knowledge graph is constructed with component ID as the core node. Based on the BIM knowledge graph, the lightweight model interaction layer enables basic interaction with the site model, model analysis and site linkage, and real-time collaborative interaction across multiple terminals. Based on the interaction processing results of the lightweight model interaction layer, the business scenario linkage layer enables closed-loop linkage for quality rectification, technical briefing, progress control, and data archiving.

2. The model interaction and data fusion method based on AI intelligent agents and multimodal deep learning according to claim 1, characterized in that, The image recognition is triggered through the following steps: Image acquisition: Take photos of on-site components and problems using a mobile terminal camera, and simultaneously collect environmental parameters, including light intensity, shooting distance, and equipment pitch angle; Image preprocessing: Gaussian filtering is used to remove image noise, and the edge contours of components are extracted using the Canny edge detection algorithm; the Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the mobile terminal camera, and the image distortion is corrected by the intrinsic parameter matrix to eliminate perspective error; BIM Component-Specific Multi-Dimensional Feature Extraction: Extracts 3 types of core features and 2 types of auxiliary features for BIM components. The feature dimensions are specifically designed for the geometric characteristics of building components to avoid redundant features. BIM Model Feature Library Pre-construction: Extract the above 5 types of features of all components in the BIM model in advance, and build a BIM component feature library according to building-floor-component type-component ID, and store it synchronously to the edge lightweight AI and the cloud core AI; the fields should at least include: component ID, geometric shape features, structural texture features, spatial location features, color features, and number of feature points. Weighted Euclidean distance feature matching algorithm: Calculate the weighted Euclidean distance between the component features in the on-site captured image and the features in the BIM component feature library, and the one with the smallest distance is the matching result; Matching result determination: If the matching result does not exceed the set first distance threshold, it is determined to be a precise match, and the component ID is directly located; if the matching result is between the first distance threshold and the second distance threshold, the Top 3 matching results are output and the user can manually select; if the matching result exceeds the second distance threshold, the matching fails, and image acquisition is repeated until it does not exceed the second distance threshold; the second distance threshold is greater than the first distance threshold.

3. The model interaction and data fusion method based on AI intelligent agents and multimodal deep learning according to claim 2, characterized in that, The three core features include geometric shape features, structural texture features, and spatial location features; The two types of auxiliary features include color features and feature point data, among which: The geometric shape features are obtained by using the RotatedRect algorithm to obtain the minimum bounding rectangle of the component, and the aspect ratio, area and perimeter of the rectangle are calculated. Seven Hu moment invariants of the contour are extracted, and the absolute value logarithmic transformation of the seven Hu moment invariants is performed to eliminate the difference in order of magnitude, which are used as geometric shape features. The structural texture features adopt the unified mode LBP operator, and extract the LBP texture histogram of the component region based on preset parameters, reducing the 256-dimensional features to 59-dimensional unified mode features as structural texture features. The spatial position feature calculation calculates the relative proportion of the center coordinates of the minimum bounding rectangle of the component in the image. Combined with the pitch angle and roll angle obtained by the gravity sensor of the shooting device, the initial position feature of the component is corrected to determine the relative position of the component in the floor, which is used as the spatial position feature. The color feature is obtained by converting the image from RGB to HSV color space, quantizing the hue channel at 36 levels, and quantizing the saturation and brightness channels at 5 levels respectively, and merging them into a 90-dimensional color histogram. The dominant hue corresponding to the peak of the histogram is taken as the color feature. The number of feature points is determined by extracting key feature points from the component region using the SIFT algorithm, and then counting the number of valid feature points.

4. The model interaction and data fusion method based on AI intelligent agents and multimodal deep learning according to claim 2, characterized in that, The weighted Euclidean distance is specifically: ,in: ； In the formula, For weighted Euclidean distance, Let represent the weight of the nth class feature, and ; This represents the normalized value of the nth class feature extracted from the image. This represents the normalized value of the nth type of feature in the BIM model feature library. For feature number, These correspond to geometric shape features, structural texture features, spatial location features, color features, and the number of feature points, respectively. The weights are adjusted using a correction factor. is the correction coefficient for the nth type of feature.

5. The model interaction and data fusion method based on AI intelligent agents and multimodal deep learning according to claim 1, characterized in that, The distributed AI parsing layer parses the on-site data, including multi-task complex instruction decomposition, multi-terminal collaborative scheduling, and multi-modal interaction intent alignment; wherein, multi-modal interaction intent alignment is achieved through the following steps: Multimodal single-channel feature extraction: For N types of modal data input synchronously by the user, single-modal feature vectors are extracted through the corresponding encoding models, and the initial intent distribution of each modality is output; Modal confidence weight calculation: Based on the recognition confidence and environmental parameters of each modality, calculate the confidence weight of each modality. In the formula, Let i be the recognition confidence level for the i-th modality. The environment adaptation coefficient for the i-th mode; Global intent fusion distribution calculation: Based on credibility weights, the initial intent distributions of all modalities are weighted and fused to obtain the global intent fusion distribution. and take each The largest category is used as the global core intent; BIM Knowledge Graph Intent Constraint Verification: Based on the global core intent, the constraint rules and entities corresponding to the intent are retrieved from the BIM knowledge graph, and the feature vector of each modality is constrained and verified to remove features that are irrelevant to the global core intent. Cross-modal intent alignment feature fusion: Using a contrastive learning approach, global core intents are used as positive samples and irrelevant intents are used as negative samples. Features from all modalities are aligned and optimized to obtain an aligned cross-modal fused feature vector. Intent parsing result output: Based on the aligned cross-modal fusion feature vector, the core slot parameters are extracted, and the final intent parsing result is output to complete the intent alignment of multimodal interaction.

6. The model interaction and data fusion method based on AI intelligent agents and multimodal deep learning according to claim 1, characterized in that, The multi-source business data is fused through the multi-source data fusion layer, and a BIM knowledge graph is constructed with component ID as the core node, specifically including: Entity Construction: Based on the BIM model analysis results, construct the core entities of the components, connect with the construction management system, acceptance specification library, and material and equipment system, extract structured data, and construct related entities; Entity alignment and disambiguation: Using component ID as the unique primary key, entities from different data sources are aligned, and issues of identical names and synonyms are eliminated; Relation extraction: Using rule matching and a fine-tuned small language model, semantic relationships between entities are extracted; Hierarchical index construction: Build a composite hierarchical index by building-floor-component type-component ID; Knowledge graph storage: The knowledge graph is stored using a MongoDB database, and a composite index of component ID, building number, and floor number is constructed.

7. The model interaction and data fusion method based on AI intelligent agents and multimodal deep learning according to claim 1, characterized in that, The basic interaction with the site model includes precise component positioning, highlighting and / or box selection marking, and visual annotation of problems; the model analysis and site linkage are achieved through the following steps: Prior information binding: In the BIM model, the actual size prior information is bound to each component, including the component's design length, width, height, and the actual spatial distance between component feature point pairs; Component location: Take photos of on-site components using a mobile terminal, and identify the corresponding BIM component ID through image feature matching; Image scale factor calculation: Extract feature point pairs in the image that correspond to prior information, calculate the pixel distance between the feature point pairs, and combine the actual spatial distance of the component to calculate the image scale factor; Actual dimensions and spacing calculation on site: Extract the edge contour of the component to be measured from the image, calculate the pixel distance by the pixel coordinates of the two endpoints of the spacing to be measured, and obtain the actual dimensions of the component and the actual spacing between the components by multiplying the pixel distance by the scale factor. Compliance comparison: Extract the design parameters, acceptance criteria and spatial relationship data of the component from the BIM structured database and knowledge graph, and compare them with the on-site measured data; Output results: If the test fails, an early warning message is generated and the component is highlighted in the lightweight model, and the problem description, test results and standard references are linked simultaneously.

8. A model interaction and data fusion system based on AI intelligent agents and multimodal deep learning, characterized in that, The system is implemented based on a multi-layer architecture, including a multimodal input layer, a distributed AI parsing layer, a multi-source data fusion layer, a lightweight model interaction layer, and a business scenario linkage layer. The system includes: The on-site data acquisition module is used to acquire on-site data through various triggering methods of the multimodal input layer, including text commands, voice commands, image recognition, on-site GPS positioning, and component scanning. The on-site data analysis module is used to analyze the on-site data using the distributed AI analysis layer, which adopts a collaborative analysis system of lightweight AI at the edge and core AI in the cloud. The graph construction module is used to construct a BIM knowledge graph based on the parsing results of the distributed AI parsing layer and the fusion of multi-source business data through the multi-source data fusion layer, with component ID as the core node. The interaction module is used to perform basic interaction with the site model, model analysis and site linkage, and real-time collaborative interaction with multiple terminals through the lightweight model interaction layer based on the BIM knowledge graph. The linkage module is used to perform quality rectification closed-loop linkage, technical briefing linkage, progress control linkage, and data archiving linkage through the business scenario linkage layer based on the interaction processing results of the lightweight model interaction layer.

9. An electronic device, characterized in that, include: At least one processor, at least one memory, and a data bus; The processor and the memory communicate with each other via the data bus. The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the model interaction and data fusion method based on AI agent and multimodal deep learning as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the model interaction and data fusion method based on AI agents and multimodal deep learning as described in any one of claims 1-7.

Images