Engineering quality analysis system and method based on artificial intelligence multimodal data fusion

Abstract

The present invention discloses an engineering quality analysis system and method based on artificial intelligence multimodal data fusion, which relates to the technical field of engineering quality analysis and includes an engineering quality analysis platform. The engineering quality analysis platform is communicatively connected to the following modules, wherein: a multi-source data fusion module is used to collect multimodal data including image data, sensor data and BIM model data from the construction site, and pre-process the multimodal data to obtain a multimodal data set; an instance segmentation and coarse classification module is used to segment quality problem instances in the image data. By fusing the multimodal information of image data, sensor data and BIM model data, the present invention can more comprehensively capture quality problems in the construction process, and use artificial intelligence algorithms to conduct in-depth analysis of the multimodal data, accurately identify and locate various quality problems, including problems that overlap or are distributed in different positions of the image, and significantly improve the recognition accuracy of quality problems.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering quality analysis, and in particular to an engineering quality analysis system and method based on artificial intelligence multimodal data fusion. Background Art

[0002] With the development of society and the acceleration of urbanization, the scale of construction projects has continued to expand, and the requirements for project quality have become increasingly higher. The quality of construction projects is directly related to the safety of people’s lives and property and social stability. Traditional quality inspection methods have gradually shown their limitations when faced with complex and diverse building structures and construction processes. With the rapid development of science and technology, technologies such as artificial intelligence, the Internet of Things, and big data have provided new ideas and methods for project quality analysis.

[0003] For example, the invention application with Chinese patent publication number CN113807722A discloses an on-site quality analysis method and system for power engineering projects based on image analysis. By obtaining multiple candidate target contours from project images with quality problems, and through unified second pixel feature extraction and pixel analysis model of the candidate target contours, at least one quality problem in the project image and at least one specific problem of at least one quality problem are analyzed.

[0004] In the existing technology, by analyzing project images with quality problems and identifying the specific problems of the quality problems, the low level of quality supervision and management of engineering projects and the difficulty in taking effective intelligent dynamic management measures to carry out multi-dimensional management of engineering project quality are solved. However, in actual engineering images, there are multiple quality problems, and the quality problems may overlap with each other or be distributed in different positions of the image, making it difficult to accurately identify and locate multiple quality problems. Summary of the Invention

[0005] The purpose of the present invention is to provide an engineering quality analysis system and method based on artificial intelligence multimodal data fusion to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, an engineering quality analysis system based on artificial intelligence multimodal data fusion includes an engineering quality analysis platform, wherein the engineering quality analysis platform is communicatively connected to the following modules, wherein:

[0008] The multi-source data fusion module is used to collect multimodal data including image data, sensor data and BIM model data from the construction site, and pre-process the multimodal data to form a unified data set to obtain a multimodal data set;

[0009] The instance segmentation and coarse classification module is used to segment the quality problem instances in the image data and perform coarse classification on the segmented quality problem instances to identify the type of quality problem;

[0010] The quality problem sub-classification module is used to further classify each quality problem instance based on the coarsely classified quality problem type and combine it with the pre-processed multimodal data to determine its specific sub-type and improve the classification accuracy;

[0011] The quality problem precise location module is used to combine image processing and spatial analysis technology to accurately locate each quality problem instance after subdivision and determine its specific location;

[0012] The result visualization module is used to conduct a comprehensive analysis of the results of sub-classification and precise positioning, generate a quality problem report, and display the quality problem report in a visual manner.

[0013] A further improvement of the technical solution of the present invention is that the method of collecting multimodal data including image data, sensor data and BIM model data from the construction site and preprocessing the multimodal data to obtain a multimodal data set includes the following steps:

[0014] Multimodal data including image data, sensor data and BIM model data are collected from the construction site and classified. For image data collection, high-definition cameras are installed at key locations on the construction site to capture images and videos of the construction process in real time. Drones are used to conduct regular inspections of the construction site and capture large-scale images of the construction progress. For sensor data collection, environmental sensors such as temperature, humidity, wind speed, and light are installed at the construction site to monitor the parameters of the construction environment in real time. Strain sensors and displacement sensors are installed at key structural parts to monitor the stress and deformation of the structure in real time. For BIM model data collection, BIM software is used to construct a detailed building information model during the engineering design phase. The model contains the building's three-dimensional geometric information, component properties, construction schedule and other information. During the construction process, the BIM model is updated in real time according to the actual construction progress and changes.

[0015] Perform preprocessing operations on the collected multimodal data, including cleaning, denoising, and standardization, to eliminate outliers and format differences;

[0016] The pre-processed image data, sensor data and BIM model data are integrated to construct a multimodal dataset with unified structure.

[0017] A further improvement of the technical solution of the present invention is that: the instance segmentation and coarse classification module includes an instance segmentation unit and a quality problem coarse classification unit;

[0018] The instance segmentation unit is used to segment and identify quality problem instances in the image data using an instance segmentation algorithm, and process overlapping quality problem areas;

[0019] The quality problem coarse classification unit is used to combine the hierarchical analysis model to coarsely classify the segmented and identified quality problem instances, identify the quality problem types, and quickly classify the quality problems into preliminary quality problem types.

[0020] A further improvement of the technical solution of the present invention is that the method of using the instance segmentation algorithm to segment and identify quality problem instances in image data and process overlapping quality problem areas includes the following steps:

[0021] In the initial stage of using the instance segmentation algorithm to segment and identify quality problem instances in image data, the input image data is preliminarily processed, including resizing the image to adapt to the model input requirements and normalizing pixel values ​​to eliminate lighting differences. A convolutional neural network (CNN) is then used to extract features from the preliminarily processed image data. Through multi-layer convolution and pooling operations, local and global features of quality problems in the image are automatically learned.

[0022] The Mask R-CNN-based instance segmentation algorithm processes the extracted local and global features. A Region Proposal Network (RPN) is used to generate candidate regions containing quality problem instances. Classification, bounding box regression, and pixel-level segmentation are then performed on the candidate regions. This generates a precise binary mask for each quality problem instance, clearly marking its location and shape in the image. Furthermore, the learned features are used to perform a preliminary classification of the instances and identify the type of quality problem.

[0023] For the overlapping quality problem areas of quality problem instances, the non-maximum suppression (NMS) algorithm is used to process them. According to the classification score and bounding box information of the instance, the instance with the highest score and no excessive overlap with other instances is screened out, and duplicate or redundant detection results are removed. The segmentation results are post-processed with morphological operations to further optimize the boundaries of the instance, making the segmentation results more accurate and clear. The precise segmentation results, type labels and location information in the image of each quality problem instance are output.

[0024] A further improvement of the technical solution of the present invention is that the method of combining the hierarchical analysis model to roughly classify the segmented and identified quality problem instances and identify the quality problem types includes the following steps:

[0025] Integrate the quality problem instance features obtained from the instance segmentation unit to form a comprehensive instance feature description, and perform preliminary screening of quality problem instances based on pre-set simple rules to quickly identify instances that meet the characteristics of common quality problem types;

[0026] Design and apply a multi-level hierarchical analysis model. Through the coarse classification layer designed in it, quality problems are divided into three types: structural, surface, and leakage. Subsequent layers are further subdivided according to actual needs, but the current focus is on the coarse classification of the first layer. The integrated quality problem instance features are input into the hierarchical analysis model. By calculating the similarity between the instance features and the feature templates of each type pre-stored in the model, the type of the instance is determined. If the similarity between the instance features and a certain type of feature template exceeds the preset similarity threshold, the instance is determined to belong to the corresponding quality problem type. The similarity is calculated using the cosine similarity method.

[0027] The rough classification results output by the hierarchical analysis model are verified. After verification, the final rough classification results of the quality problems are output. The output content includes the unique identifier of the quality problem instance, the type of quality problem to which it belongs, and the confidence information of the classification.

[0028] A further improvement of the technical solution of the present invention is that the method of further classifying each quality problem instance based on the roughly classified quality problem type in combination with the pre-processed multimodal data to determine its specific subtype includes the following steps:

[0029] After obtaining the coarse classification results from the quality problem coarse classification module, we combine multimodal data to enhance the features of each quality problem instance. We fuse the binary mask and bounding box in the image data with the sensor data and semantic information in the BIM model. We use the binary mask and bounding box as spatial constraints, map the sensor data and semantic information to the corresponding image area, and generate a comprehensive multimodal feature description.

[0030] The fused multimodal features are input into a pre-trained deep learning model for fine-grained classification. The Transformer architecture is used to further subdivide the features into specific subtypes, and the specific subtype label and confidence score of each instance are output.

[0031] The sub-classification results are verified and optimized. The dataset is divided into training and test sets using the holdout method to ensure that the data distribution of the training and test sets is similar. By comparing the actual labels of the test set with the model predicted labels, the classification accuracy, recall rate and F1 score indicators are calculated to evaluate the classification effect, and then the sub-classification results of each quality problem instance are output, including the unique identifier of the instance, the specific sub-type label and the corresponding confidence level.

[0032] A further improvement to the technical solution of the present invention is that the method of inputting the fused multimodal features into a pre-trained deep learning model for subclassification, further subdividing into specific subtypes using a Transformer architecture, and outputting the specific subtype label and confidence of each instance includes the following steps:

[0033] The Transformer architecture was used to build a deep learning model. The number of layers and hidden units in the model was determined based on the number of quality problem types after coarse classification. The model weight parameters were initialized and transfer learning was performed using the parameters of the pre-trained model to accelerate model convergence. The binary mask, bounding box, sensor data, and semantic information in the BIM model were arranged in order. The image-related binary mask and bounding box features were arranged first, followed by the sensor data features, and finally the BIM semantic information features. This formed a continuous structured input sequence as the input to the Transformer model.

[0034] The serialized multimodal features are input into the pre-trained Transformer model. The Transformer's self-attention mechanism is used to automatically learn the association and importance between features, assign different weights to each feature, and highlight features that have a significant impact on classification.

[0035] After feature learning by the Transformer model, each quality problem instance is sub-classified through the fully connected layer and the Softmax function, and then each quality problem instance is sub-classified and assigned to a specific sub-type label. At the same time, the confidence of each classification result is calculated based on the probability distribution output by the model.

[0036] A further improvement of the technical solution of the present invention is that the method of combining image processing and spatial analysis technology to accurately locate each quality problem instance after subdivision and determine its specific location includes the following steps:

[0037] After obtaining the sub-classification results from the quality problem sub-classification module, the image data of each quality problem instance is further processed. The CornerNet key point detection algorithm is used to identify key points related to the quality problem by predicting the heat map, embedding vector, and offset of the corner points. The coordinate position of the key points in the image data is then determined, including the key point sequence number and the horizontal and vertical coordinate values ​​in the image. This preliminarily determines the scope of the quality problem in the image space and the location of key features.

[0038] Using camera calibration technology, a mapping relationship between the image coordinate system and the real-space coordinate system is established. The conversion parameters between the image coordinates and the real-space coordinates are calculated using known calibration points. The horizontal and vertical coordinate values ​​of the identified key points in the image are mapped to the real-space coordinate system using the conversion parameters. The preliminary position of the key points of the quality problem in real space is obtained. The positioning results are optimized in combination with sensor data. The sensor data is fused with the image positioning results. The Kalman filter data fusion algorithm is used to eliminate the error of single sensor positioning, improve the positioning accuracy of the quality problem in space, determine a more accurate three-dimensional spatial position, and obtain precise positioning results.

[0039] The precise positioning results are verified and optimized. The positioning accuracy is evaluated by comparing the positioning results with the actual measurement data. If the positioning deviation is found to exceed the preset deviation threshold, the cause is analyzed and the mapping algorithm is adjusted. After verification and optimization, the precise positioning results of each quality problem instance are output.

[0040] A further improvement of the technical solution of the present invention is that the method of comprehensively analyzing the results of subdivision and precise positioning, generating a quality problem report, and visually displaying the quality problem report includes the following steps:

[0041] After obtaining the processing results from the quality problem sub-classification module and the precise location module, a comprehensive analysis is conducted on the sub-classification results and precise location results of each quality problem instance, and the information is integrated into a unified data structure to ensure that the classification and location information of each quality problem instance can be correlated with each other. At the same time, combined with the component information in the BIM model, the quality problem instance is mapped to a specific building component;

[0042] Generate a quality problem report based on the integrated data. Utilize the 3D visualization function of the BIM model to intuitively display the classification and location results of quality problem instances in the 3D model. At the same time, generate a detailed quality problem report, including information such as the type, location, confidence level, and location accuracy of the quality problem.

[0043] The generated quality problem report is presented to the user in an interactive manner. At the same time, a user feedback mechanism is provided to allow the user to mark or correct the quality problems in the report to further optimize the analysis and visualization effects of the system.

[0044] In a second aspect, a method for analyzing engineering quality based on artificial intelligence multimodal data fusion is implemented based on the aforementioned engineering quality analysis system based on artificial intelligence multimodal data fusion, and includes the following steps:

[0045] Collect image data, sensor data, and BIM model data from the construction site, classify and aggregate them, and then perform cleaning, denoising, and standardization to form a unified data set;

[0046] The instance segmentation algorithm is used to segment and identify quality problem instances in image data, and the hierarchical analysis model is combined for rough classification to identify the type of quality problem;

[0047] Combine multimodal data to enhance the features of quality problem instances after rough classification, input pre-trained deep learning models for fine classification, determine specific subtypes, and improve classification accuracy;

[0048] Image processing and spatial analysis technology are used to accurately locate the quality problems after subdivision and determine their three-dimensional spatial location;

[0049] Comprehensively analyze the sub-classification and precise positioning results to generate a quality problem report, and use the 3D visualization function of the BIM model to display the results and provide a user feedback mechanism.

[0050] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art:

[0051] The present invention provides an engineering quality analysis system and method based on artificial intelligence multimodal data fusion. By fusing multimodal information of image data, sensor data and BIM model data, it can more comprehensively capture quality problems in the construction process, and use artificial intelligence algorithms to conduct in-depth analysis of multimodal data to accurately identify and locate various quality problems, including problems that overlap or are distributed in different positions in the image, significantly improving the accuracy of identifying quality problems.

[0052] The present invention provides an engineering quality analysis system and method based on artificial intelligence multimodal data fusion. By adopting a hierarchical analysis model and a deep learning algorithm, quality problems are coarsely and finely classified, and combined with spatial analysis technology to achieve precise positioning, the system can more accurately determine the type and location of quality problems, improve the efficiency of engineering management, and enable managers to respond to and handle quality problems more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0054] Figure 1 This is a schematic diagram of the workflow of the engineering quality analysis system based on artificial intelligence multimodal data fusion of the present invention;

[0055] Figure 2The figure is a flow chart of the engineering quality analysis method based on artificial intelligence multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] Example 1, as Figure 1 As shown, the present invention provides an engineering quality analysis system based on artificial intelligence multimodal data fusion, including an engineering quality analysis platform, which is communicatively connected to the following modules, wherein:

[0058] The multi-source data fusion module is used to collect multimodal data including image data, sensor data and BIM model data from the construction site, and pre-process the multimodal data to form a unified data set to obtain a multimodal data set. Multimodal data including image data, sensor data and BIM model data are collected from the construction site and classified and collected. For image data collection, high-definition cameras are installed at key locations of the construction site (concrete pouring area, steel structure welding area, material stacking area, etc.) to capture images and videos of the construction process in real time. UAVs are used to conduct regular inspections of the construction site to capture large-scale construction Progress images are used to check whether the overall layout of the construction area and the construction progress are in line with the plan. The collected image data is classified and stored according to the construction stage, construction area, and quality problem type. For sensor data collection, environmental sensors such as temperature, humidity, wind speed, and light are installed at the construction site to monitor the parameters of the construction environment in real time. Strain sensors and displacement sensors are installed at key structural parts to monitor the stress and deformation of the structure in real time. The collected sensor data is classified and stored according to sensor type, monitoring object, and construction stage. For BIM model data collection, during the engineering design phase, BIM software is used to construct a detailed building information model. The model contains the building's three-dimensional geometric information, component properties, construction schedule, and other information. During the construction process, the BIM model is updated in real time according to the actual construction progress and changes. The BIM model data is classified and stored according to the type of building component, construction stage, and functional area. The collected multimodal data is preprocessed, including cleaning, denoising, and standardization, to eliminate outliers and format differences. Among them, by cleaning the image data, quality problems such as blur, overexposure, and underexposure in the image are removed. The sensor data is cleaned to remove outliers in the sensor data. Clean the BIM model data, check whether the component attributes in the BIM model are complete and accurate, and whether there is duplicate or missing component information. De-noise the image data and use image denoising algorithms (wavelet transform denoising, non-local mean denoising, etc.) to remove noise interference in the image. De-noise the sensor data and use filtering algorithms for denoising. De-noise the BIM model data. In the BIM model, some minor geometric deviations caused by modeling errors or data import errors are corrected through model optimization algorithms. Standardize the image data and normalize the pixel values ​​of the image data to [0,1], standardize sensor data, unify the dimensions of sensor data, standardize BIM model data, standardize the component attributes in the BIM model, ensure the compatibility of BIM model data with other data, integrate pre-processed image data, sensor data and BIM model data, and build a multimodal data set with unified structure. Among them, artificial intelligence algorithm (spatial transformation network) is used to assist spatial alignment. Through geographic information system (GIS) technology, the image data of the construction site is matched with the three-dimensional geographic coordinates in the BIM model to ensure that the objects in the image are consistent with the components in the BIM model in spatial position. According to the installation location information of the sensor, it is associated with the corresponding component in the BIM model, and the data of different modalities are synchronized in time. The image data and sensor data are aligned in time based on the image shooting time and the acquisition timestamp of the sensor data using the time series analysis algorithm. This ensures that when analyzing the construction status at a certain moment, the image data and sensor data can reflect the situation at the same time point. The association relationship between different modal data is established, and the quality problems detected in the image are associated with the component attributes in the BIM model using association rule mining. At the same time, the quality problems are further analyzed in combination with the sensor data to mine potential quality problem patterns.

[0059] The instance segmentation and coarse classification module is used to segment the quality problem instances in the image data and coarsely classify the segmented quality problem instances to identify the type of quality problem. The instance segmentation and coarse classification module includes an instance segmentation unit and a quality problem coarse classification unit;

[0060] Among them, the instance segmentation unit is used to segment and identify quality problem instances in image data using the instance segmentation algorithm, and process overlapping quality problem areas. In the initial stage of the process of segmenting and identifying quality problem instances in image data using the instance segmentation algorithm, the input image data is preliminarily processed, including adjusting the image size to adapt to the model input requirements and normalizing the pixel values ​​to eliminate the illumination difference. Then, the convolutional neural network (CNN) is used to extract features from the preliminarily processed image data. Through multi-layer convolution and pooling operations, the local and global features of quality problems in the image are automatically learned, and the image data is converted into a high-dimensional feature vector. The Mask-based R-CNN's instance segmentation algorithm processes the extracted local and global features, generates candidate regions containing quality problem instances through the region proposal network (RPN), and then performs classification, bounding box regression, and pixel-level segmentation on the candidate regions. It then generates an accurate binary mask for each quality problem instance, clearly marking the location and shape of the instance in the image. At the same time, it preliminarily classifies the instances based on the learned features and identifies the type of quality problem. For overlapping quality problem areas of quality problem instances, it uses the non-maximum suppression (NMS) algorithm to process them. Based on the instance classification score and bounding box information, it selects the instances with the highest scores that do not overlap with other instances excessively, removes duplicate or redundant detection results, and performs morphological operations on the segmentation results to further optimize the instance boundaries, making the segmentation results more accurate and clear. It outputs the accurate segmentation results, type labels, and location information of each quality problem instance in the image.

[0061] The quality problem coarse classification unit is used to combine the hierarchical analysis model to coarsely classify the segmented and identified quality problem instances, identify the quality problem type, and quickly classify the quality problems into preliminary quality problem types. The quality problem instance features obtained from the instance segmentation unit are integrated to form a comprehensive instance feature description, and the quality problem instances are preliminarily screened according to pre-set simple rules. For example, an area threshold is set to judge instances with an area less than a certain value as noise or irrelevant interference and directly exclude them. According to the color feature, if the instance color is too different from the color of common quality problems, it is also eliminated. Instances that meet the characteristics of common quality problem types are quickly locked in, and invalid instances that obviously do not meet any type of characteristics are removed to improve the efficiency and accuracy of subsequent classification. A multi-level hierarchical analysis model is designed and applied. Through the coarse classification layer designed therein, quality problems are divided into three types: structural, surface and leakage. Subsequent layers are further subdivided according to actual needs, but the current focus is on In the first-level coarse classification, the integrated quality problem instance features are input into the hierarchical analysis model. The type of the instance is determined by calculating the similarity between the instance features and the feature templates of various types pre-stored in the model. If the similarity between the instance features and a certain type of feature template exceeds the preset similarity threshold, the instance is determined to belong to the corresponding quality problem type. The similarity is calculated using the cosine similarity method. The coarse classification results output by the hierarchical analysis model are verified. The cross-validation method is used to divide the data set into multiple subsets. One of the subsets is used as the test set and the remaining subsets are used as the training set in turn. The experiment is repeated many times. The accuracy and recall rate of the classification results are statistically analyzed to check the accuracy of the classification results. If many classification errors are found, the reasons are analyzed and the model is adjusted and optimized. After verification, the final quality problem coarse classification results are output. The output content includes the unique identifier of the quality problem instance, the type of quality problem to which it belongs, and the confidence information of the classification.

[0062] The quality problem sub-classification module is used to further classify each quality problem instance based on the coarsely classified quality problem type and combine it with the pre-processed multimodal data to determine its specific sub-type and improve the classification accuracy;

[0063] The quality problem precise positioning module is used to combine image processing and spatial analysis technology to accurately locate each quality problem instance after subdivision, determine its specific location, and achieve precise spatial positioning of quality problems;

[0064] The result visualization module is used to conduct a comprehensive analysis of the results of sub-classification and precise positioning, generate a quality problem report, and display the quality problem report in a visual manner.

[0065] Example 2, as Figure 1As shown, based on Example 1, the present invention provides a technical solution: Preferably, for the quality problem type after rough classification, each quality problem instance is further classified in combination with the preprocessed multimodal data, and the steps of determining its specific subtype include:

[0066] After obtaining the coarse classification results from the quality problem coarse classification module, the multimodal data is combined to enhance the features of each quality problem instance, and the binary mask and bounding box in the image data are fused with the sensor data and the semantic information in the BIM model. The binary mask and bounding box are used as spatial constraints, and the sensor data and semantic information are mapped to the corresponding image area to generate a comprehensive multimodal feature description. The fused multimodal features are input into the pre-trained deep learning model for fine classification. The Transformer architecture is used to further subdivide them into specific subtypes, and the specific subtype label and confidence of each instance are output. The fine classification results are verified and optimized, and the holdout method is used. Divide the dataset into a training set and a test set, ensuring that the data distribution of the training set and the test set is similar. Compare the actual labels of the test set with the model's predicted labels, and calculate the classification accuracy, recall, and F1 score indicators to evaluate the classification effect. Accuracy measures the correctness of the model's overall classification, recall reflects the model's ability to find all positive examples, and the F1 score is the harmonic mean of accuracy and recall. Comprehensively evaluate the model's performance. If the classification accuracy of certain subtypes is low, analyze the cause and adjust the model parameters or optimize the feature extraction process. Then, output the detailed classification results for each quality issue instance, including the instance's unique identifier, specific subtype label, and corresponding confidence level.

[0067] In addition, the fused multimodal features are input into a pre-trained deep learning model for sub-classification, further subdivided into specific subtypes using a Transformer architecture, and the specific subtype label and confidence of each instance are output. The steps of the method include:

[0068] The Transformer architecture is used to build a deep learning model. According to the number of quality problem types after coarse classification, the parameters of the model's number of layers and hidden units are determined. At the same time, the model's weight parameters are initialized, and the parameters of the pre-trained model are used for transfer learning to accelerate model convergence. The binary mask, bounding box, sensor data, and semantic information in the BIM model are arranged in order. The image-related binary mask and bounding box features are arranged first, followed by the sensor data features, and finally the BIM semantic information features to form a continuous structured input sequence as the input of the Transformer model. The serialized multimodal features are input into the pre-trained Transformer model. The Transformer's self-attention mechanism automatically learns the associations and importance between features, assigning different weights to each feature to highlight those that have a significant impact on classification. After feature learning by the Transformer model, each quality issue instance is subclassified using a fully connected layer and a Softmax function. The fully connected layer maps the features to output dimensions corresponding to the number of subtypes. The Softmax function converts the output into a probability distribution, further subclassifying each quality issue instance and assigning it to a specific subtype label. The confidence level of each classification result is calculated based on the probability distribution output by the model. The higher the confidence level, the more reliable the model's judgment that the instance belongs to that subtype.

[0069] Combining image processing and spatial analysis technology, each quality problem instance after subdivision is accurately located. The method for determining its specific location includes the following steps:

[0070] After obtaining the sub-classification results from the quality problem sub-classification module, the image data of each quality problem instance is further processed. The CornerNet key point detection algorithm is used to identify the key points related to the quality problem by predicting the heat map, embedding vector and offset of the corner points, and then determine the coordinate position of the key points in the image data, including the sequence number of the key points and the horizontal and vertical coordinate values ​​in the image. The scope of the quality problem in the image space and the key feature position are preliminarily clarified. The camera calibration technology is used to establish a mapping relationship between the image coordinate system and the actual space coordinate system. The conversion parameters of the image coordinates and the actual space coordinates are calculated through the known calibration points. The horizontal and vertical coordinate values ​​of the identified key points in the image are mapped to the actual space coordinate system through the conversion parameters, and the key points of the quality problem are obtained. The initial position in the real space is obtained, and the positioning results are optimized in combination with the sensor data. The sensor data is fused with the image positioning results. The error of single sensor positioning is eliminated through the Kalman filter data fusion algorithm, the positioning accuracy of the quality problem in space is improved, and a more accurate three-dimensional spatial position is determined. The precise positioning results are obtained, and the precise positioning results are verified and optimized. The positioning accuracy is evaluated by comparing the positioning results with the actual measurement data. If the positioning deviation is found to exceed the preset deviation threshold, the cause is analyzed and the mapping algorithm is adjusted. After verification and optimization, the precise positioning results of each quality problem instance are output, including the unique identifier of the instance and the specific coordinates in the three-dimensional space. Among them, the preset deviation threshold is the maximum allowable error range between the actual measurement data and the positioning result;

[0071] The method for comprehensively analyzing the results of sub-classification and precise positioning, generating a quality problem report, and visually displaying the quality problem report includes the following steps:

[0072] After obtaining the processing results from the quality problem sub-classification module and the precise positioning module, a comprehensive analysis is performed on the sub-classification results and precise positioning results of each quality problem instance, and each information is integrated into a unified data structure to ensure that the classification and positioning information of each quality problem instance can be correlated with each other. At the same time, combined with the component information in the BIM model, the quality problem instance is mapped to a specific building component. Based on the integrated data, a quality problem report is generated. The three-dimensional visualization function of the BIM model is used to display the classification and positioning results of the quality problem instance in an intuitive manner in the three-dimensional model. Different colors or icons are used to distinguish the type of quality problem, and arrows or annotations are used to display the specific location of the quality problem. At the same time, a detailed quality problem report is generated, including information such as the type, location, confidence, positioning accuracy, etc. of the quality problem. The generated quality problem report is displayed to the user in an interactive manner. At the same time, a user feedback mechanism is provided to allow the user to mark or correct the quality problems in the report to further optimize the analysis and visualization effects of the system.

[0073] Example 3, as Figure 1 、 Figure 2 As shown, based on Examples 1-2, the present invention further provides an engineering quality analysis method based on artificial intelligence multimodal data fusion, which is implemented based on an engineering quality analysis system based on artificial intelligence multimodal data fusion, and includes the following steps:

[0074] Collect image data, sensor data, and BIM model data from the construction site, classify and aggregate them, and then perform cleaning, denoising, and standardization to form a unified data set;

[0075] The instance segmentation algorithm is used to segment and identify quality problem instances in image data, and the hierarchical analysis model is combined for rough classification to identify the type of quality problem;

[0076] Combine multimodal data to enhance the features of quality problem instances after rough classification, input pre-trained deep learning models for fine classification, determine specific subtypes, and improve classification accuracy;

[0077] Image processing and spatial analysis technology are used to accurately locate the quality problems after subdivision and determine their three-dimensional spatial location;

[0078] Comprehensively analyze the sub-classification and precise positioning results to generate a quality problem report, and use the 3D visualization function of the BIM model to display the results and provide a user feedback mechanism.

[0079] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. The engineering quality analysis system based on artificial intelligence multimodal data fusion includes an engineering quality analysis platform, which is characterized by: The engineering quality analysis platform is connected to the following modules: The multi-source data fusion module is used to collect multimodal data including image data, sensor data and BIM model data from the construction site, and pre-process the multimodal data to obtain a multimodal data set; The instance segmentation and coarse classification module is used to segment the quality problem instances in the image data and perform coarse classification on the segmented quality problem instances to identify the type of quality problem; The quality problem sub-classification module is used to further classify each quality problem instance based on the coarsely classified quality problem type and combine it with the pre-processed multimodal data to determine its specific sub-type. It includes the following steps: After obtaining the coarse classification results, the multimodal data is combined to enhance the features of each quality problem instance. The binary mask and bounding box in the image data are fused with the sensor data and semantic information in the BIM model. The binary mask and bounding box are used as spatial constraints, and the sensor data and semantic information are mapped to the corresponding image area to generate a multimodal feature description. The fused multimodal features are input into a pre-trained deep learning model for fine classification, and further subdivided into specific subtypes using the Transformer architecture, and the specific subtype label and confidence of each instance are output. The method includes the following steps: selecting the Transformer architecture to build a deep learning model, determining the parameters of the model's number of layers and hidden units based on the number of quality problem types after coarse classification, initializing the model's weight parameters, and using the parameters of the pre-trained model for transfer learning to accelerate model convergence. The binary mask, bounding box, sensor data, and semantic information in the BIM model are arranged in sequence to form a continuous structured input sequence as the input of the Transformer model; the serialized multimodal features are input into the pre-trained Transformer model, and the Transformer's self-attention mechanism is used to automatically learn the association and importance between features, assigning different weights to each feature; after feature learning by the Transformer model, each quality problem instance is finely classified through a fully connected layer and a Softmax function, and then each quality problem instance is finely classified and assigned to a specific subtype label. At the same time, the confidence of each classification result is calculated based on the probability distribution output by the model; Verify and optimize the sub-classification results. Use the holdout method to divide the dataset into training and test sets to ensure that the data distribution of the training and test sets is similar. By comparing the actual labels of the test set with the model-predicted labels, calculate the classification accuracy, recall rate, and F1 score indicators to evaluate the classification effect. Then, output the sub-classification results of each quality issue instance, including the instance's unique identifier, specific sub-type label, and corresponding confidence level. The quality problem precise location module is used to combine image processing and spatial analysis technology to accurately locate each quality problem instance after subdivision and determine its specific location; The result visualization module is used to conduct a comprehensive analysis of the results of sub-classification and precise positioning, generate a quality problem report, and display the quality problem report in a visual manner.

2. The engineering quality analysis system based on artificial intelligence multimodal data fusion according to claim 1 is characterized by: The method of collecting multimodal data including image data, sensor data and BIM model data from the construction site and preprocessing the multimodal data to obtain a multimodal data set includes the following steps: Collect multimodal data including image data, sensor data and BIM model data from the construction site and classify and aggregate them; Perform preprocessing operations on the collected multimodal data, including cleaning, denoising, and standardization, to eliminate outliers and format differences; The pre-processed image data, sensor data and BIM model data are integrated to construct a multimodal dataset with unified structure.

3. The engineering quality analysis system based on artificial intelligence multimodal data fusion according to claim 1 is characterized by: The instance segmentation and coarse classification module includes an instance segmentation unit and a quality problem coarse classification unit; The instance segmentation unit is used to segment and identify quality problem instances in the image data using an instance segmentation algorithm, and process overlapping quality problem areas; The quality problem coarse classification unit is used to combine the hierarchical analysis model to perform coarse classification on the segmented and identified quality problem instances and identify the quality problem type.

4. The engineering quality analysis system based on artificial intelligence multimodal data fusion according to claim 3 is characterized by: The method for segmenting and identifying quality problem instances in image data using an instance segmentation algorithm and processing overlapping quality problem areas comprises the following steps: In the initial stage of segmenting and identifying quality problem instances in image data using the instance segmentation algorithm, the input image data is preliminarily processed, including resizing the image to fit the model input requirements and normalizing pixel values ​​to eliminate lighting differences. A convolutional neural network is then used to extract features from the preliminarily processed image data, automatically learning the local and global features of quality problems in the image. The Mask R-CNN-based instance segmentation algorithm processes the extracted local and global features. A region proposal network is used to generate candidate regions containing quality problem instances. Classification, bounding box regression, and pixel-level segmentation are then performed on the candidate regions. This generates a precise binary mask for each quality problem instance, clearly marking its location and shape in the image. Furthermore, the learned features are used to perform a preliminary classification of the instances and identify the type of quality problem. For the overlapping quality problem areas of quality problem instances, the non-maximum suppression algorithm is used to process them. According to the classification score and bounding box information of the instance, the instance with the highest score and no excessive overlap with other instances is screened out, and the segmentation results are post-processed with morphological operations to further optimize the instance boundaries. The precise segmentation results, type labels and location information in the image of each quality problem instance are output.

5. The engineering quality analysis system based on artificial intelligence multimodal data fusion according to claim 4 is characterized by: The method of combining the hierarchical analysis model to roughly classify the segmented and identified quality problem instances and identify the quality problem types includes the following steps: Integrate the quality problem instance features obtained from the instance segmentation unit to form a comprehensive instance feature description, and perform preliminary screening of quality problem instances based on pre-set simple rules to quickly identify instances that meet the characteristics of common quality problem types; A multi-level hierarchical analysis model was designed and applied. Through the coarse classification layer, quality issues were classified into three types: structural, surface, and leakage. The integrated quality issue instance features were input into the hierarchical analysis model. The instance type was determined by calculating the similarity between the instance features and the feature templates of each type pre-stored in the model. If the similarity between the instance features and a feature template of a certain type exceeded a preset similarity threshold, the instance was determined to belong to the corresponding quality issue type. The rough classification results output by the hierarchical analysis model are verified. After verification, the final rough classification results of the quality problems are output. The output content includes the unique identifier of the quality problem instance, the type of quality problem to which it belongs, and the confidence information of the classification.

6. The engineering quality analysis system based on artificial intelligence multimodal data fusion according to claim 1 is characterized by: The method for accurately locating each quality problem instance after subdivision by combining image processing and spatial analysis technology to determine its specific location includes the following steps: After obtaining the sub-classification results from the quality problem sub-classification module, the image data of each quality problem instance is further processed. The CornerNet key point detection algorithm is used to identify key points related to the quality problem by predicting the heat map, embedding vector, and offset of the corner points. The coordinate position of the key points in the image data is then determined, including the key point sequence number and the horizontal and vertical coordinate values ​​in the image. This preliminarily determines the scope of the quality problem in the image space and the location of key features. Using camera calibration technology, a mapping relationship between the image coordinate system and the real-space coordinate system is established. The conversion parameters between the image coordinates and the real-space coordinates are calculated using known calibration points. The horizontal and vertical coordinate values ​​of the identified key points in the image are mapped to the real-space coordinate system using the conversion parameters. The preliminary position of the key points of the quality problem in real space is obtained. The positioning results are then optimized in combination with sensor data. The sensor data and image positioning results are then fused to obtain accurate positioning results. The precise positioning results are verified and optimized. The positioning accuracy is evaluated by comparing the positioning results with the actual measurement data. If the positioning deviation is found to exceed the preset deviation threshold, the cause is analyzed and the mapping algorithm is adjusted. After verification and optimization, the precise positioning results of each quality problem instance are output.

7. The engineering quality analysis system based on artificial intelligence multimodal data fusion according to claim 6 is characterized by: The method of comprehensively analyzing the results of subdivision and precise positioning, generating a quality problem report, and visually displaying the quality problem report includes the following steps: After obtaining the processing results from the quality problem sub-classification module and the precise positioning module, a comprehensive analysis is conducted on the sub-classification results and precise positioning results of each quality problem instance, and the information is integrated into a unified data structure. At the same time, combined with the component information in the BIM model, the quality problem instance is mapped to the specific building component; Generate a quality problem report based on the integrated data. Utilize the 3D visualization function of the BIM model to intuitively display the classification and location results of quality problem instances in the 3D model. At the same time, generate a detailed quality problem report. The generated quality problem report is presented to the user in an interactive manner. At the same time, a user feedback mechanism is provided to allow the user to mark or correct the quality problems in the report.

8. A method for analyzing engineering quality based on artificial intelligence multimodal data fusion, implemented based on the system for analyzing engineering quality based on artificial intelligence multimodal data fusion according to any one of claims 1 to 7, characterized in that: The following steps are involved: Collect image data, sensor data, and BIM model data from the construction site, classify and aggregate them, and then perform cleaning, denoising, and standardization to form a unified data set; The instance segmentation algorithm is used to segment and identify quality problem instances in image data, and the hierarchical analysis model is combined for rough classification to identify the type of quality problem; Combine multimodal data to enhance the features of quality problem instances after rough classification, input them into the pre-trained deep learning model for fine classification, and determine the specific subtype; Image processing and spatial analysis technology are used to accurately locate the quality problems after subdivision and determine their three-dimensional spatial location; Comprehensively analyze the sub-classification and precise positioning results to generate a quality problem report, and use the 3D visualization function of the BIM model to display the results and provide a user feedback mechanism.

Images