Fabricated building component intelligent generation and real-time detection method and system based on multi-modal AI

Abstract

The invention relates to a fabricated building component intelligent generation and real-time detection method and system based on multi-modal AI. The method comprises the steps that according to a building specification text and a preset BIM model library, cross-modal semantic alignment processing is conducted through a CLIP model, a text-image feature mapping table is generated, analysis is conducted based on a natural language instruction, component design parameters are obtained, and a compliance BIM component model is generated; acquiring three-dimensional point cloud data of an actual installation component, performing error detection processing on the compliant BIM component model through an improved YOLO network, and outputting a detection result; and performing incremental updating processing on the CLIP model according to a detection result to obtain an updated CLIP model. According to the method, through multi-modal semantic alignment, natural language driven design, high-precision construction detection and closed-loop self-optimization, the intelligence of assembly type building component design, the accuracy of construction detection and the adaptability of a model are improved, and effective technical support is provided for efficient construction and quality control of an assembly type building.

Description

Technical Field

[0001] The present invention belongs to the intersection of artificial intelligence technology and building industrialization, and in particular relates to a method and system for intelligent generation and real-time detection of prefabricated building components based on multimodal AI. Background Art

[0002] With the advancement of technologies integrating artificial intelligence and building industrialization, prefabricated construction and AI-based design and inspection technologies have emerged. Prefabricated buildings are highly efficient and environmentally friendly, significantly shortening construction cycles. AI-based design and inspection technologies, on the other hand, leverage AI algorithms to assist in design and visual algorithms to conduct construction inspections, further enhancing the intelligence of building design and construction processes.

[0003] However, traditional technologies often rely on designers to manually translate building code requirements into corresponding BIM (Building Information Modeling) design parameters during the design phase of prefabricated building components. This makes it difficult to automatically link requirements with regulations such as the "Technical Code for Prefabricated Concrete Structures" (JGJ1-2014). This lack of automated cross-modal association capabilities leads to inefficient and error-prone designs. For example, when generating prefabricated shear walls that meet specific seismic requirements, manual code lookup, reinforcement ratio calculation, and node configuration design are required. Furthermore, there is no automatic mapping mechanism for converting natural language instructions into design parameters, requiring manual interpretation of instructions before parameter setting. Furthermore, during construction inspection, visual algorithms such as OpenCV are often used to detect component installation errors. However, the false detection rate is high when identifying irregularly shaped components (such as sandwich wall panels with embedded pipelines). This is particularly true when occluding 3D models, making it difficult to automatically detect conflicts between steel trusses and pipelines. Furthermore, BIM reviews rely on manual view switching, making it impossible to identify construction feasibility issues in real time. This leads to high rates of errors, omissions, and collisions in prefabricated components and increases rework costs. Summary of the Invention

[0004] Based on this, it is necessary to provide a multimodal AI-based intelligent generation and real-time detection method and system for prefabricated building components to address the above technical problems, so as to further improve the design efficiency of prefabricated building components and the accuracy of actual construction detection, thereby enhancing the automation level of the entire process of building design and construction.

[0005] In the first aspect, this application provides a method for intelligent generation and real-time detection of prefabricated building components based on multimodal AI, including:

[0006] Based on the building code text and the preset BIM model library, cross-modal semantic alignment processing is performed through the CLIP model to generate a text-image feature mapping table. The text-image feature mapping table includes mapping rules between building code attributes and BIM component parameters.

[0007] Parse the component design parameters based on natural language instructions and text-image feature mapping tables to generate a compliant BIM component model;

[0008] Acquire 3D point cloud data of components actually installed during construction, perform error detection on the compliant BIM component model using an improved YOLO network, and output detection results, including collision detection results and position offset quantification results.

[0009] Based on the detection results, the CLIP model is incrementally updated to obtain the updated CLIP model.

[0010] In one embodiment, a cross-modal semantic alignment process is performed using a CLIP model based on the building code text and a preset BIM model library to generate a text-image feature mapping table, including:

[0011] Construct a multimodal knowledge graph based on building code texts and a pre-set BIM model library; the building code texts include building component material parameters, node structures, and seismic resistance levels, while the pre-set BIM model library includes component dimensions, embedded parts locations, and collision history data;

[0012] Based on the text data in the multimodal knowledge graph and the BIM model rendering image, the CLIP model is used for comparative learning to generate aligned text-image feature pairs;

[0013] Generate a text-image feature mapping table based on the aligned text-image feature pairs.

[0014] In one embodiment, the comparison function of the CLIP model is:

[0015]

[0016] Among them, L contrast is the contrast function, T i is the text feature vector generated by encoding the i-th building specification text through the BERT word segmenter, I j To preset BIM model library with T i The image feature vector extracted by ResNet-50 from the multi-view rendering image of the corresponding component, τ is the temperature coefficient, and N is the total number of samples processed in one training.

[0017] In one embodiment, parsing is performed based on natural language instructions and a text-image feature mapping table to obtain design parameters and generate a compliant BIM component model, including:

[0018] According to natural language instructions, semantic keywords are extracted through the CLIP model to generate a set of design constraints, which includes building code attribute keywords and parameter constraints.

[0019] Using building code attribute keywords, the corresponding BIM component parameters are matched from the text-image feature mapping table to form a parameter mapping set;

[0020] Based on the logical verification of parameter mapping set and parameter constraint conditions, component design parameters are generated. Component design parameters include material type, component size, embedded parts positioning coordinates and connection node form;

[0021] Call the preset BIM parametric modeling engine to generate a three-dimensional BIM component model based on the component design parameters, perform compliance verification based on the collision history data in the preset BIM model library, and output a compliant BIM component model.

[0022] In one embodiment, three-dimensional point cloud data of components actually installed during construction is obtained, and error detection processing is performed on the compliant BIM component model using an improved YOLO network. The detection results are output, including collision detection results and position offset quantification results, including:

[0023] Perform voxelization and outlier filtering preprocessing on the 3D point cloud data. Use the ICP rigid registration algorithm based on 3D spatial position relationships to align the coordinate system of the 3D point cloud data with the local coordinate system of the compliant BIM component model to generate a standardized point cloud dataset.

[0024] The standardized point cloud dataset is input into the improved YOLOv7 network, which includes a dynamic weight allocation module and a sub-pixel contour regression branch, where:

[0025] The dynamic weight allocation module is based on the 3D spatial attention mechanism. It calculates the cosine similarity between the query vector of the occluded component and the key vectors of the surrounding components, generates a visibility probability value, and dynamically adjusts the convolutional layer weights.

[0026] The sub-pixel contour regression branch performs 3D feature processing on the standardized point cloud dataset and outputs the sub-pixel 3D coordinates and Euler angle pose parameters of the actual installation component bounding box.

[0027] Compare the sub-pixel 3D coordinates with the theoretical coordinates of the compliant BIM component model, calculate the position offset and the rotation angle deviation around the axis, and generate a quantitative result of the position offset;

[0028] The visibility probability value is compared with the preset occlusion threshold, and the spatial overlapping volume of the bounding box is calculated to determine the component collision type and generate a collision detection result. The collision detection result includes the three-dimensional coordinates of the collision location, the BIM number of the component involved, and the collision severity level.

[0029] In one embodiment, the method further includes training and optimizing the improved YOLOv7 network using an occlusion-aware loss function, wherein the occlusion-aware loss function includes a bounding box regression loss, an offset error loss, and a visibility probability prediction loss, and the visibility probability prediction loss is calculated based on a visibility probability value generated by a dynamic weight allocation module.

[0030] In one embodiment, incremental updating of the CLIP model is performed based on the detection results to obtain an updated CLIP model, including:

[0031] Filter out construction error data based on the detection results, where the construction error data includes position deviation exceeding limit data and / or collision conflict data;

[0032] Based on the construction error data and the compliant BIM component model, an incremental training dataset is constructed, where:

[0033] Generate revised design parameter text based on construction error data combined with the design parameter text of the compliant BIM component model through the CLIP model;

[0034] Using construction error data, we can filter out erroneous components from compliant BIM component models and generate a corrected training image set by rendering the erroneous components from multiple perspectives.

[0035] According to the revised design parameter text and the revised training image set, a revised text-image feature pair is generated to obtain an incremental training data set;

[0036] Based on the incremental training dataset, the CLIP model is incrementally trained to obtain an updated CLIP model.

[0037] Secondly, this application also provides a multimodal AI-based intelligent generation and real-time detection system for prefabricated building components, including:

[0038] A cross-modal semantic alignment module is used to perform cross-modal semantic alignment processing based on the building code text and the preset BIM model library through the CLIP model to generate a text-image feature mapping table. The text-image feature mapping table includes mapping rules between building code attributes and BIM component parameters;

[0039] The BIM component generation module is used to parse the natural language instructions and text-image feature mapping table to obtain component design parameters and generate a compliant BIM component model;

[0040] The real-time construction detection module is used to obtain 3D point cloud data of components actually installed during construction. It then uses an improved YOLO network to perform error detection on compliant BIM component models and outputs detection results, including collision detection results and position offset quantification results.

[0041] The closed-loop feedback optimization module is used to incrementally update the CLIP model based on the detection results to obtain an updated CLIP model.

[0042] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the first aspect when executing the computer program.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the first aspect when executed by a processor.

[0044] The above-mentioned multimodal AI-based intelligent generation and real-time detection method and system for prefabricated building components uses the CLIP model to perform cross-modal semantic alignment processing based on building code text and a pre-set BIM model library to generate a text-image feature map. This effectively overcomes the drawback of traditional prefabricated building component design, which often makes it difficult to accurately associate building codes with component parameters. It establishes clear mapping rules between building code attributes and BIM component parameters, thus providing an accurate basis for the subsequent acquisition of component design parameters. Secondly, component design parameters are obtained by parsing natural language instructions and the text-image feature map. This not only achieves automatic conversion between natural language and design parameters, but also avoids the errors and workload of manual instruction interpretation. Furthermore, by acquiring 3D point cloud data of components actually installed during construction and using an improved YOLO network to perform error detection on compliant BIM component models, this method can output detailed detection information, including collision detection results and position offset quantification results. This effectively improves the accuracy and efficiency of construction error detection and enables timely detection of collisions and position offset issues during construction. Finally, the CLIP model is incrementally updated through detection results, which enables the model to continuously learn new construction conditions and error information, improving the adaptability and accuracy of the model in different construction scenarios.

[0045] Compared with traditional methods, this method improves the intelligence of prefabricated building component design, the accuracy of construction inspection and the adaptability of the model through cross-modal semantic alignment, natural language parsing, improved detection network and incremental update of the model, providing effective technical support for the efficient construction and quality control of prefabricated buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of a multimodal AI-based intelligent generation and real-time detection method for prefabricated building components provided by an exemplary embodiment of the present invention;

[0048] Figure 2 A schematic structural diagram of a multimodal AI-based intelligent generation and real-time detection system for prefabricated building components, provided as an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] In one embodiment, Figure 1 As shown, a method for intelligent generation and real-time detection of prefabricated building components based on multimodal AI is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0051] S101: Based on the building code text and the preset BIM model library, cross-modal semantic alignment processing is performed through the CLIP model to generate a text-image feature mapping table. The text-image feature mapping table includes mapping rules between building code attributes and BIM component parameters.

[0052] Specifically, building specification text and BIM model library are two different modal data. Building specification text describes the design requirements of building components in natural language. Building specification attributes such as fire protection level, seismic resistance level, reinforcement ratio, and node structure can be extracted from a large number of specification documents such as "Assembly Building Evaluation Standard" and "Assembly Concrete Building Technical Standard GB / T 51231". The BIM model library stores three-dimensional models of building components in image and parameterized form. In order to associate the data of these two modalities, this embodiment uses the CLIP model (Contrastive Language-Image Pretraining, CLIP) to perform cross-modal semantic alignment. The CLIP model can map text and image features to the same semantic space, thereby generating a text-image feature mapping table. This mapping table defines the mapping rules between building specification attributes such as size, material, and installation method and BIM component parameters. Through this mapping table, it is possible to automatically understand the building specification text and convert it into corresponding component parameters in the BIM model library, providing a basis for subsequent component design.

[0053] S102: Analyze the natural language instructions and the text-image feature mapping table to obtain component design parameters and generate a compliant BIM component model.

[0054] Specifically, during the design phase, users can describe their desired prefabricated building components using natural language commands, such as "Generate a prestressed double-T slab with a span of 6m that meets the fire protection requirements of GB / T 51231." By parsing the semantic information in the command, key content can be extracted, such as "span 6m," "prestressed double-T slab," and "GB / T 51231 fire protection requirements." The text-image feature mapping table generated by S101 can then be used to find the BIM component parameters corresponding to these key content. For example, based on the "GB / T 51231 fire protection requirements," parameters such as slab thickness, prestressed tendon spacing, and end anchorage length that meet the fire protection level can be found. Combined with the "span 6m," other relevant dimensional parameters can be further determined. Based on these parameters, a BIM component model that complies with building regulations can be generated. Furthermore, during the generation process, a lightweight YOLOv7-tiny model can be used to perform real-time verification of the generated BIM component model. The YOLOv7-tiny model can quickly identify potential problems in the model, such as collisions between components and mismatches in size or parameters, ensuring the feasibility of the generated BIM component model in practical applications.

[0055] S103: Acquire the three-dimensional point cloud data of the components actually installed during the construction process, perform error detection processing on the compliant BIM component model through the improved YOLO network, and output the detection results, which include collision detection results and position offset quantification results.

[0056] Specifically, during the construction process, the actual installed components may deviate from the design model due to construction errors. In this embodiment, 3D point cloud data from the construction process can be acquired using an AR device. This data accurately records the actual spatial position and shape information of the components at the construction site. This data can be compared with the compliant BIM component model generated in S102. To improve detection efficiency and accuracy, an improved YOLO network that incorporates a dynamic weight allocation mechanism can be used for error detection. Illustratively, during error detection, the improved YOLO network can preprocess the acquired 3D point cloud data. For example, it can remove noise points through statistical filtering, radius filtering, and other methods to further improve data quality. The network can then use the dynamic weight allocation mechanism to extract and match features between the compliant BIM component model and the actual 3D point cloud data. This dynamic weight allocation mechanism can dynamically adjust the weights of different features during the matching process based on their importance. For example, the network can automatically assign higher weights to key features such as connection points and stress points, while assigning relatively lower weights to less important surface features. This mechanism can more accurately focus on the key factors affecting component performance and improve detection accuracy.

[0057] Specifically, during the feature matching process, the network compares the feature points of each component point by point. For collision detection, the network analyzes the spatial relationship between points in the 3D point cloud data to determine whether there is overlap between the point clouds of different components. Once an overlapping area is found, it is marked as a potential collision area and the severity and scope of the collision are further calculated. For example, the severity of the collision can be quantified by calculating parameters such as the volume and area of the overlapping area, providing a detailed basis for subsequent processing. For position offset quantification, the network calculates the spatial distance between the feature points of the component in the actual 3D point cloud data and the corresponding feature points in the compliant BIM component model. By performing statistical analysis on the distances between multiple feature points, the overall position offset of the component is determined. After completing collision detection and position offset quantification, the improved YOLO network can output the detection results in an intuitive manner. For example, the collision detection results can be visually marked in the 3D model, with different colors or symbols indicating the severity of the collision. This allows construction personnel to quickly locate the problem and take appropriate adjustments, such as fine-tuning or reinstalling the component, to ensure that construction quality meets design requirements.

[0058] S104: performing incremental updating processing on the CLIP model based on the detection results to obtain an updated CLIP model.

[0059] Specifically, incremental updating means optimizing the model through a small amount of training data while retaining the original model capabilities. For example, if the test results show that there is a deviation between certain specification attributes and actual construction data, this data can be fed back into the training set of the CLIP model. The CLIP model can perform incremental update processing based on the newly added data, readjust the parameters, and further optimize the alignment effect between the specification text and the BIM model image features, so that the mapping of specification attributes and BIM component parameters is more accurate, thereby obtaining an updated CLIP model. In the subsequent component generation process, the updated CLIP model can take into account the problems that have occurred before and generate a BIM component model that is more in line with actual construction needs, thereby avoiding similar errors and forming a virtuous closed loop of "generation-detection-optimization", and continuously improving the design and construction quality of prefabricated building components.

[0060] In this method, cross-modal semantic alignment of building code text with a pre-set BIM model library effectively eliminates semantic differences between different modal data. Furthermore, by generating a text-image feature map, precise matching of building code attributes with BIM component parameters is achieved, providing an accurate basis for the subsequent acquisition of component design parameters. Secondly, by parsing natural language instructions and feature maps, compliant BIM component models can be rapidly generated, avoiding the errors and inefficiencies associated with manual code interpretation in traditional design, significantly improving design efficiency and accuracy. Furthermore, an improved YOLO network is used to detect errors in 3D point cloud data during construction, providing real-time output of collision detection results and position offset quantification. This addresses the data lag and difficulty in quantifying errors in traditional detection methods, providing a more accurate basis for construction quality control. Finally, incremental updates to the CLIP model based on detection results enable timely response to new developments and error information during construction, further enhancing adaptability and detection capabilities to new design requirements and reducing safety hazards during construction.

[0061] In one embodiment, a cross-modal semantic alignment process is performed based on the building code text and the preset BIM model library through the CLIP model to generate a text-image feature mapping table, including:

[0062] Construct a multimodal knowledge graph based on building code texts and a pre-set BIM model library; the building code texts include building component material parameters, node structures, and seismic resistance levels, while the pre-set BIM model library includes component dimensions, embedded parts locations, and collision history data;

[0063] Based on the text data in the multimodal knowledge graph and the BIM model rendering image, the CLIP model is used for comparative learning to generate aligned text-image feature pairs;

[0064] Generate a text-image feature mapping table based on the aligned text-image feature pairs.

[0065] Specifically, when constructing a multimodal knowledge graph, information from building code text and BIM model libraries can be organized and integrated. This involves extracting concepts and relationships from the text, as well as attributes and associations from the BIM model, and representing them in a graph. For example, "steel strength grade" can be linked to components using that grade of steel in the relevant BIM model, ultimately forming a rich multimodal knowledge graph that provides a foundation for subsequent cross-modal semantic alignment. Furthermore, after obtaining the multimodal knowledge graph, images can be rendered using the text data and BIM models, and comparative learning can be performed using the CLIP model. For text data, the BERT tokenizer can be used. The BERT tokenizer breaks down sentences in building code text into tokens and encodes these tokens into computer-processable text feature vectors. For example, "The seismic resistance grade of this building component is level three" can be converted into a specific text feature vector after processing by the BERT tokenizer. This vector contains the semantic information of each token in the sentence. For BIM models, multi-view rendering can be performed to generate a series of images from different angles. The rendered image is then subjected to feature extraction using a ResNet-50 model to obtain the corresponding image feature vector. The CLIP model can then use a contrast function for comparative learning. This contrast function calculates the similarity between the text feature vector and the image feature vector to determine whether the two match.

[0066] Schematically, the comparison function of the CLIP model is:

[0067]

[0068] Among them, L contrast is the contrast function, T i is the text feature vector generated by encoding the i-th building specification text through the BERT word segmenter, I j To preset BIM model library with T i The image feature vector extracted by ResNet-50 from the multi-view rendering image of the corresponding component, τ is the temperature coefficient, and N is the total number of samples processed in one training.

[0069] In the above formula, the temperature coefficient plays a regulatory role, which can control the distribution range of the similarity score, making the model more stable during the learning process. In one training session, the CLIP model can process N samples. By continuously adjusting the model parameters, the model can learn the semantic association between text and image, and finally obtain the aligned text-image feature pairs, and then generate a text-image feature mapping table. This mapping table records the mapping rules between building specification attributes and BIM component parameters. Through this mapping table, in subsequent applications, when a building specification text is input, the corresponding BIM component parameters can be quickly found, thereby realizing the generation of a BIM component model that meets the requirements according to the building specification, or performing compliance checks on existing BIM models.

[0070] In one embodiment, parsing is performed based on natural language instructions and text-image feature mapping tables to obtain design parameters and generate a compliant BIM component model, including:

[0071] According to natural language instructions, semantic keywords are extracted through the CLIP model to generate a set of design constraints, which includes building code attribute keywords and parameter constraints.

[0072] Using building code attribute keywords, the corresponding BIM component parameters are matched from the text-image feature mapping table to form a parameter mapping set;

[0073] Based on the logical verification of parameter mapping set and parameter constraint conditions, component design parameters are generated. Component design parameters include material type, component size, embedded parts positioning coordinates and connection node form;

[0074] Call the preset BIM parametric modeling engine to generate a three-dimensional BIM component model based on the component design parameters, perform compliance verification based on the collision history data in the preset BIM model library, and output a compliant BIM component model.

[0075] Specifically, designers or other relevant personnel can provide design instructions in natural language, such as "Design a precast beam that meets the seismic resistance level of Class II and the fire protection rating of Class A, with a length of 6 meters." When the CLIP model receives this natural language instruction, it performs operations such as word segmentation to break the sentence into meaningful lexical units. Its internal semantic analysis mechanism extracts key semantic information, such as "seismic resistance level of Class II," "fire protection rating of Class A," "precast beam," and "beam length of 6 meters," thereby forming a set of design constraints. Building code attribute keywords represent the regulatory requirements that must be adhered to in the building design, while parameter constraints specify the specific component dimensions or grade requirements. The building code attribute keywords are then accurately matched against the text-to-image feature mapping table, and all matching BIM component parameters are collected to form a parameter mapping set. This set contains a series of component parameters related to the design requirements, providing a rich data source for the subsequent generation of accurate component design parameters.

[0076] After obtaining the parameter mapping set and parameter constraints, they need to be logically checked to ensure that the generated component design parameters meet the requirements of the building code and are logically reasonable and feasible. Schematically, you can check whether the parameters in the parameter mapping set meet the parameter constraints. For example, if the parameter constraint requires the beam length to be 6 meters, and the beam length parameter obtained from the parameter mapping set does not meet this requirement, it needs to be adjusted or further screened. At the same time, it will also check whether the logical relationship between the various parameters is reasonable, such as whether the reinforcement ratio and concrete strength grade of the component match each other, whether they can meet the requirements of the seismic grade, etc., and then generate the final component design parameters. The design parameters can include parameters such as material type, component size, embedded parts positioning coordinates and connection node form, which describe the various characteristics of the component in detail and provide a specific data basis for generating an accurate BIM component model.

[0077] Based on the component design parameters, a preset BIM parametric modeling engine can be called to generate a three-dimensional BIM component model. A BIM parametric modeling engine is a tool specifically designed to create three-dimensional models based on component parameters. It can accurately construct a component's three-dimensional model based on input parameters such as material type, component size, embedded part positioning coordinates, and connection node form. Furthermore, after the model is generated, compliance verification can be performed using collision history data from a preset BIM model library. This collision history data records collisions between components in previous projects and related information. The newly generated three-dimensional BIM component model can be compared and analyzed with this collision history data to check for potential collision issues. For example, it can be checked whether the position of embedded parts will conflict with other components, and whether the component's size will cause collisions with the surrounding environment during installation. If the model is verified to have no compliance issues such as collisions, it can be output as a compliant BIM component model.

[0078] In one embodiment, three-dimensional point cloud data of components actually installed during construction is obtained, and error detection processing is performed on the compliant BIM component model using an improved YOLO network. The detection results are output. The detection results include collision detection results and position offset quantification results, including:

[0079] Perform voxelization and outlier filtering preprocessing on the 3D point cloud data. Use the ICP rigid registration algorithm based on 3D spatial position relationships to align the coordinate system of the 3D point cloud data with the local coordinate system of the compliant BIM component model to generate a standardized point cloud dataset.

[0080] The standardized point cloud dataset is input into the improved YOLOv7 network, which includes a dynamic weight allocation module and a sub-pixel contour regression branch, where:

[0081] The dynamic weight allocation module is based on the 3D spatial attention mechanism. It calculates the cosine similarity between the query vector of the occluded component and the key vectors of the surrounding components, generates a visibility probability value, and dynamically adjusts the convolutional layer weights.

[0082] The sub-pixel contour regression branch performs 3D feature processing on the standardized point cloud dataset and outputs the sub-pixel 3D coordinates and Euler angle pose parameters of the actual installation component bounding box.

[0083] Compare the sub-pixel 3D coordinates with the theoretical coordinates of the compliant BIM component model, calculate the position offset and the rotation angle deviation around the axis, and generate a quantitative result of the position offset;

[0084] The visibility probability value is compared with the preset occlusion threshold, and the spatial overlapping volume of the bounding box is calculated to determine the component collision type and generate a collision detection result. The collision detection result includes the three-dimensional coordinates of the collision location, the BIM number of the component involved, and the collision severity level.

[0085] Specifically, voxelization divides the 3D space into small voxels and assigns the 3D point cloud data to these voxels, reducing data density and volume while also smoothing the data to a certain extent. Outlier filtering identifies and removes points that significantly deviate from the surrounding point cloud data to improve data quality. Furthermore, to accurately compare the 3D point cloud data of the actual installed components with the compliant BIM component model, an ICP rigid registration algorithm based on 3D spatial positional relationships can be used for coordinate alignment. This algorithm, through continuous iteration, finds the optimal rotation and translation transformation to align the coordinate system of the 3D point cloud data with the local coordinate system of the compliant BIM component model as closely as possible, thereby generating a standardized point cloud dataset. This dataset has a more regular data structure and a coordinate system consistent with the compliant BIM component model, providing a sound data foundation for subsequent error detection. The standardized point cloud dataset can then be input into a modified YOLOv7 network for processing. However, at the construction site, components may occlude each other, further impacting the accuracy of error detection. In this embodiment, a dynamic weight allocation module generates a query vector for each obscured component based on a three-dimensional spatial attention mechanism. This query vector contains the component's feature information. Key vectors of surrounding components are also obtained, reflecting their characteristics. Subsequently, a visibility probability value is calculated by calculating the cosine similarity between the query vector of the obscured component and the key vectors of surrounding components. This value indicates the likelihood that the obscured component can be accurately detected in the current environment.

[0086] Based on the visibility probability value, the dynamic weight allocation module can dynamically adjust the weights of the network's convolutional layers. When the visibility probability is low, it means that the component is severely obscured and difficult to detect. In this case, the weights of the relevant convolutional layers can be increased, allowing the network to pay more attention to the characteristics of this component, thereby improving detection accuracy. The sub-pixel contour regression branch is mainly used to perform three-dimensional feature processing on the standardized point cloud dataset. That is, through a series of convolution, pooling and other operations, it extracts the bounding box information of the actual installed component. Unlike traditional detection methods that can only obtain integer-level coordinates, the sub-pixel contour regression branch can output the sub-pixel three-dimensional coordinates and Euler angle pose parameters of the actual installed component's bounding box. These sub-pixel three-dimensional coordinates are more precise and can more accurately describe the component's position. The Euler angle pose parameters represent the component's rotation state in three-dimensional space, including the rotation angles around the X, Y, and Z axes.

[0087] Specifically, after obtaining the sub-pixel 3D coordinates of the actual installed component's bounding box, they can be compared with the theoretical coordinates of the compliant BIM component model. By calculating the difference between the two, the position offset is obtained. This offset reflects the deviation between the component's actual position and the designed position in the X, Y, and Z directions. Furthermore, by comparing the Euler angle pose parameters reflected by the sub-pixel 3D coordinates with the theoretical pose parameters of the compliant BIM component model, the axial rotation angle deviation can be calculated—the difference between the component's actual rotation angle and the designed rotation angle along the X, Y, and Z axes. Combining the position offset and axial rotation angle deviation generates a quantitative position offset result. This result intuitively reflects the degree of deviation between the actual installed component's position and pose and the design requirements, helping construction personnel determine whether adjustments are necessary. Furthermore, the visibility probability value obtained by the dynamic weight allocation module can be compared with a preset occlusion threshold. If the visibility probability value is lower than the occlusion threshold, it indicates a high probability of occlusion, posing a collision risk. The spatial overlap volume of the bounding box can also be calculated by analyzing the overlap in 3D space between the bounding boxes of the actual installed component and the compliant BIM component model. Based on the size of the overlap volume and the visibility probability value, the component collision type can be further determined, such as minor, moderate, or severe.

[0088] In one embodiment, the method further includes training and optimizing the improved YOLOv7 network using an occlusion-aware loss function, wherein the occlusion-aware loss function includes a bounding box regression loss, an offset error loss, and a visibility probability prediction loss, and the visibility probability prediction loss is calculated based on the visibility probability value generated by the dynamic weight allocation module.

[0089] Specifically, the improved YOLOv7 network can be trained using a large amount of training data with real-world labels. This data can include 3D point cloud data of actual installed components and their corresponding real-world bounding box information, position offset information, and visibility information. In each training iteration, the improved YOLOv7 network processes the input training data to predict the component's bounding box, offset, and visibility probability. Then, based on the predicted results and the real-world labels, the bounding box regression loss, offset error loss, and visibility probability prediction loss are calculated, and the three are summed to obtain the total occlusion perception loss. Based on this total occlusion perception loss, algorithms such as stochastic gradient descent can be used to adjust the network's parameters. This update aims to reduce the loss, enabling the network to more accurately predict the component's bounding box, offset, and visibility probability in subsequent predictions. Through multiple training iterations and continuous adjustment of network parameters, the occlusion perception loss is gradually reduced, and the network's performance is continuously improved. Ultimately, the trained and optimized improved YOLOv7 network can more accurately handle various complex situations in actual construction component error detection, providing a strong guarantee for the quality and efficiency of building construction.

[0090] In one embodiment, the CLIP model is incrementally updated based on the detection results to obtain an updated CLIP model, including:

[0091] Filter out construction error data based on the detection results, where the construction error data includes position deviation exceeding limit data and / or collision conflict data;

[0092] Based on the construction error data and the compliant BIM component model, an incremental training dataset is constructed, where:

[0093] Generate revised design parameter text based on construction error data combined with the design parameter text of the compliant BIM component model through the CLIP model;

[0094] Using construction error data, we can filter out erroneous components from compliant BIM component models and generate a corrected training image set by rendering the erroneous components from multiple perspectives.

[0095] According to the revised design parameter text and the revised training image set, a revised text-image feature pair is generated to obtain an incremental training data set;

[0096] Based on the incremental training dataset, the CLIP model is incrementally trained to obtain an updated CLIP model.

[0097] Specifically, construction error data can be filtered from the inspection results to perform targeted updates on the CLIP model. Construction error data primarily includes position deviation data and / or collision data. Position deviation data refers to deviations between the actual installed component position and the designed position exceeding the specified range. For example, in a building, the horizontal deviation of a precast beam installation position exceeds the design tolerance, or the vertical height deviation is excessive. Collision data refers to collisions between components during installation, such as pipes colliding with structural beams. This construction error data reflects the discrepancy between actual construction and design. The design parameter text of the compliant BIM component model details the various design requirements and standards for the component. These two can be combined for further analysis using the CLIP model. The CLIP model, with its powerful semantic understanding and feature alignment capabilities, can analyze the impact of construction error data on design parameters and modify the design parameter text accordingly. For example, if position deviation data indicates that the installation position of a component needs to be adjusted, the CLIP model can modify the component's position parameter description in the design parameter text accordingly, generating a revised design parameter text.

[0098] Specifically, construction error data can be used to screen out erroneous components within a compliant BIM component model. These erroneous components are then rendered from multiple perspectives, generating a series of images from different angles to comprehensively display the component's appearance and structural information. For example, a component experiencing a collision can be rendered from multiple angles, such as the front, side, and top, to obtain images of the component from different perspectives. This generates a corrected training image set, which reflects the actual state of the component after accounting for construction errors. The corrected design parameter text is then matched with the corrected training image set. Leveraging the cross-modal feature alignment capabilities of the CLIP model, corrected text-image feature pairs are generated. These feature pairs contain the corrected text features and the corresponding image features, establishing new semantic associations between the two. All generated corrected text-image feature pairs are then integrated to generate an incremental training dataset. This dataset, which incorporates new information after accounting for construction errors, provides richer and more accurate data for further training of the CLIP model. Based on this constructed incremental training dataset, an existing CLIP model can be incrementally trained. During training, the CLIP model continuously learns the semantic associations between corrected text-image feature pairs in incremental training datasets, adjusting its parameters to better adapt to new data. This further improves the accuracy of semantic alignment between building code text and BIM model images, ultimately resulting in an updated CLIP model. In subsequent applications, this updated model can more accurately generate BIM component models that meet actual construction requirements based on building code text, further improving quality control during the construction process.

[0099] Based on the same inventive concept, Figure 2 As shown, the embodiment of the present application also provides a multimodal AI-based prefabricated building component intelligent generation and real-time detection system 200 for implementing the multimodal AI-based prefabricated building component intelligent generation and real-time detection method involved above. The implementation solution provided by this system is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal AI-based prefabricated building component intelligent generation and real-time detection system provided below can be found in the above-mentioned limitations of the multimodal AI-based prefabricated building component intelligent generation and real-time detection method, and will not be repeated here. The system includes:

[0100] A cross-modal semantic alignment module 201 is configured to perform cross-modal semantic alignment processing based on the building specification text and the preset BIM model library using the CLIP model to generate a text-image feature mapping table, wherein the text-image feature mapping table includes mapping rules between building specification attributes and BIM component parameters;

[0101] A BIM component generation module 202 is configured to parse the natural language instructions and the text-image feature mapping table to obtain component design parameters and generate a compliant BIM component model;

[0102] The real-time construction detection module 203 is used to obtain the three-dimensional point cloud data of the components actually installed during the construction process, and perform error detection processing on the compliant BIM component model through the improved YOLO network, and output the detection results, which include collision detection results and position offset quantification results;

[0103] The closed-loop feedback optimization module 204 is configured to perform incremental updating processing on the CLIP model based on the detection results to obtain an updated CLIP model.

[0104] In the above-mentioned multimodal AI-based intelligent generation and real-time detection system 200 for prefabricated building components, the cross-modal semantic alignment module 201 can perform cross-modal semantic alignment processing based on the building specification text and the preset BIM model library through the CLIP model, effectively solving the problem of inaccurate semantic docking between the specification text and the BIM model, and improving the automation and intelligence of the design and generation process. Moreover, through the feature mapping table, the attributes of the building specification can be accurately matched with the parameters of the BIM component, thereby providing a clear and reliable parameter basis for subsequent component design and generation. The BIM component generation module 202 can perform analysis based on natural language instructions and text-image feature mapping tables, quickly respond to different building requirements, and automatically generate BIM component models that meet the design specifications, greatly improving the efficiency of component design and generation, and reducing errors and time delays caused by human intervention. The real-time construction detection module 203 obtains the three-dimensional point cloud data of the actual installed components and uses the improved YOLO network for error detection. It can detect the errors of BIM components during the construction process in real time, thereby promptly discovering potential problems in construction and correcting them, effectively avoiding building quality problems caused by component deviations, reducing rework and costs, and further improving the accuracy and reliability of construction detection.

[0105] The closed-loop feedback optimization module 204 incrementally updates the CLIP model based on the inspection results, generating an updated CLIP model. This process promptly responds to new developments and errors during construction, continuously optimizing the text-image feature mapping table, enhancing the system's adaptability to different construction scenarios, and continuously improving the quality of prefabricated building component design and construction inspection.

[0106] In an exemplary embodiment, the present invention further provides a computer device comprising a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multimodal AI-based intelligent generation and real-time detection method for prefabricated building components described herein. A multi-core processor is preferred to improve the system's parallel processing capabilities. The memory provides sufficient temporary storage space to support program execution and data processing. The memory capacity should be large enough to accommodate large amounts of supply information and computing tasks.

[0107] In an exemplary embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal AI-based method for intelligent generation and real-time detection of prefabricated building components of the present application. The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. The random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

[0108] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A method for intelligent generation and real-time detection of prefabricated building components based on multimodal AI, characterized by: The method comprises: Based on the building code text and the preset BIM model library, cross-modal semantic alignment processing is performed through the CLIP model to generate a text-image feature mapping table, which includes mapping rules between building code attributes and BIM component parameters; Parsing the natural language instructions and the text-image feature mapping table to obtain component design parameters and generate a compliant BIM component model; Acquire three-dimensional point cloud data of components actually installed during construction, perform error detection on the compliant BIM component model using an improved YOLO network, and output detection results, including collision detection results and position offset quantification results; The CLIP model is incrementally updated based on the detection results to obtain an updated CLIP model.

2. The method according to claim 1, characterized in that The cross-modal semantic alignment process is performed based on the building specification text and the preset BIM model library through the CLIP model to generate a text-image feature mapping table, including: Constructing a multimodal knowledge graph based on the building specification text and the preset BIM model library; the building specification text includes building component material parameters, node structure and seismic resistance level; and the preset BIM model library includes component size, embedded part location and collision history data; Based on the text data in the multimodal knowledge graph and the BIM model rendering image, comparative learning is performed through the CLIP model to generate aligned text-image feature pairs; The text-image feature mapping table is generated according to the aligned text-image feature pairs.

3. The method according to claim 2, characterized in that The contrast function of the CLIP model is: Among them, L contrast is the contrast function, T i is the text feature vector generated by encoding the i-th building specification text through the BERT word segmenter, I j For the preset BIM model library i The image feature vector extracted by ResNet-50 from the multi-view rendering image of the corresponding component, τ is the temperature coefficient, and N is the total number of samples processed in one training.

4. The method according to claim 1, wherein The parsing based on the natural language instruction and the text-image feature mapping table to obtain design parameters and generate a compliant BIM component model includes: Extracting semantic keywords through the CLIP model according to the natural language instructions to generate a design constraint set, wherein the design constraint set includes building code attribute keywords and parameter constraints; Matching corresponding BIM component parameters from the text-image feature mapping table using the building specification attribute keywords to form a parameter mapping set; Performing a logic check based on the parameter mapping set and the parameter constraint conditions to generate component design parameters, wherein the component design parameters include material type, component size, embedded part positioning coordinates, and connection node form; A preset BIM parametric modeling engine is called to generate a three-dimensional BIM component model according to the component design parameters, and compliance verification is performed based on the collision history data in the preset BIM model library to output the compliant BIM component model.

5. The method according to claim 1, wherein The three-dimensional point cloud data of the components actually installed during the construction process is obtained, and the error detection processing of the compliant BIM component model is performed through the improved YOLO network, and the detection results are output. The detection results include collision detection results and position offset quantification results, including: Perform voxelization and outlier filtering preprocessing on the three-dimensional point cloud data, and use an ICP rigid registration algorithm based on three-dimensional spatial position relationships to align the coordinate system of the three-dimensional point cloud data with the local coordinate system of the compliant BIM component model to generate a standardized point cloud dataset; The standardized point cloud dataset is input into the improved YOLOv7 network, wherein the improved YOLOv7 network includes a dynamic weight allocation module and a sub-pixel contour regression branch, wherein: The dynamic weight allocation module is based on the three-dimensional spatial attention mechanism, which calculates the cosine similarity between the query vector of the occluded component and the key vectors of the surrounding components, generates a visibility probability value and dynamically adjusts the convolution layer weights; The sub-pixel level contour regression branch performs three-dimensional feature processing on the standardized point cloud data set, and outputs the sub-pixel level three-dimensional coordinates and Euler angle posture parameters of the actual installation component bounding box; Comparing the sub-pixel three-dimensional coordinates with the theoretical coordinates of the compliant BIM component model, calculating the position offset and the rotation angle deviation around the axis, and generating a position offset quantification result; The visibility probability value is compared with the preset occlusion threshold, and is calculated in combination with the spatial overlapping volume of the bounding box to determine the component collision type and generate the collision detection result. The collision detection result includes the three-dimensional coordinates of the collision position, the BIM number of the component involved, and the collision severity level.

6. The method according to claim 5, characterized in that The method also includes training and optimizing the improved YOLOv7 network using an occlusion-aware loss function, wherein the occlusion-aware loss function includes a bounding box regression loss, an offset error loss, and the visibility probability prediction loss, and the visibility probability prediction loss is calculated based on the visibility probability value generated by the dynamic weight allocation module.

7. The method according to claim 1, characterized in that The incremental updating of the CLIP model according to the detection result to obtain an updated CLIP model includes: Filtering construction error data according to the detection results, wherein the construction error data includes position deviation exceeding limit data and / or collision conflict data; An incremental training dataset is constructed based on the construction error data and the compliant BIM component model, wherein: Based on the construction error data and the design parameter text of the compliant BIM component model, a revised design parameter text is generated through the CLIP model; Screening out erroneous components in the compliant BIM component model using the construction error data, and generating a corrected training image set by performing multi-view rendering on the erroneous components; Generating a corrected text-image feature pair according to the corrected design parameter text and the corrected training image set to obtain the incremental training data set; Based on the incremental training data set, the CLIP model is incrementally trained to obtain the updated CLIP model.

8. Intelligent generation and real-time detection system for prefabricated building components based on multimodal AI, characterized by: The system comprises: A cross-modal semantic alignment module is used to perform cross-modal semantic alignment processing based on the building specification text and the preset BIM model library through the CLIP model to generate a text-image feature mapping table. The text-image feature mapping table includes mapping rules between building specification attributes and BIM component parameters; A BIM component generation module is used to parse the natural language instructions and the text-image feature mapping table to obtain component design parameters and generate a compliant BIM component model; A real-time construction detection module is used to obtain 3D point cloud data of components actually installed during construction, perform error detection on the compliant BIM component model using an improved YOLO network, and output detection results, including collision detection results and position offset quantification results; The closed-loop feedback optimization module is used to perform incremental updating processing on the CLIP model based on the detection results to obtain an updated CLIP model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images