Building material defect intelligent identification method based on visual inspection

Abstract

The invention discloses a building material defect intelligent identification method based on visual inspection. The method comprises the following steps: step 1, collecting and preprocessing original multi-modal data of a building material; 2, carrying out space and time feature extraction on the building material data set based on a double-flow space-time mechanism; step 3, carrying out joint reasoning on the spectral information and the spatio-temporal feature vector set; 4, inputting the building material defect feature set into the improved PinSAGE model for spatial modeling to obtain a spatial dependency graph; 5, performing defect reasoning on building material defects according to the spatial dependency relationship graph; 6, generating a defect detection strategy of the building material through a rule engine; and step 7, positioning defects in the building material according to the defect detection strategy. Through the improved PinSAGE model, the building material defect detection precision and prediction capability are improved.

Description

Technical Field

[0001] This invention relates to the field of geographic information technology, and in particular to an intelligent identification method for defects in building materials based on visual inspection. Background Technology

[0002] With the rapid development of the construction industry, the quality and safety of building materials are receiving increasing attention, making defect detection of building materials a crucial step in ensuring the structural safety and service life of buildings. Traditional methods for detecting defects in building materials typically rely on manual inspection and experience-based judgment, such as visual inspection, manual measurement, and physical testing. These methods are not only inefficient and costly, but also cannot meet the demands of large-scale and high-precision testing. Especially in complex building environments, manual inspection is easily affected by environmental conditions, the experience of the inspectors, and subjective factors, limiting the accuracy and comprehensiveness of defect detection.

[0003] In recent years, with the development of computer vision technology, image processing-based methods for detecting defects in building materials have been gradually applied. Existing technologies typically acquire surface or thermal images of building materials using sensors such as RGB cameras and infrared thermal imagers, and then perform feature extraction and defect identification through image processing algorithms. These methods have the advantage of high automation, reducing human intervention. However, most existing image processing methods rely on only a single type of image data, such as RGB images or infrared thermal images, making it difficult to handle minute cracks and deep defects in complex backgrounds. Especially when the surface texture of building materials is complex or contains multiple defect types, traditional methods are prone to misjudgments and missed detections. Furthermore, existing technologies also have shortcomings in spatiotemporal feature extraction and defect prediction. Traditional defect detection methods often fail to consider the evolution of defects in building materials. Defects in building materials are not static but constantly change with time, environment, and usage. Traditional methods fail to effectively capture the trend of defect expansion. For example, cracks may gradually expand over a long period, and corrosion may be exacerbated by changes in the external environment, but traditional methods lack the ability to predict these dynamic changes and cannot provide early warnings of potential safety hazards.

[0004] Therefore, how to provide a visual inspection-based intelligent identification method for building material defects is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an intelligent method for identifying defects in building materials based on visual inspection. This invention combines multimodal data acquisition, spatiotemporal feature extraction, and hyperspectral imaging technology, overcoming the limitations of traditional defect detection methods. Through a dual-stream spatiotemporal mechanism, an improved PinSAGE model, and a defect evolution prediction model, this invention achieves precise location, dynamic tracking, and future evolution prediction of defects in building materials. This method not only improves the detection accuracy of micro-cracks and deep defects but also provides forward-looking support for repair work, significantly enhancing the comprehensiveness and efficiency of building material quality inspection.

[0006] A method for intelligent identification of defects in building materials based on visual inspection according to an embodiment of the present invention includes the following steps: Step 1: Collect raw multimodal data of building materials through multiple sensors, and preprocess the raw multimodal data to generate a building material dataset; Step 2: Based on the dual-stream spatiotemporal mechanism, spatial and temporal features are extracted from the building material dataset to obtain a spatiotemporal feature vector set; Step 3: Obtain the spectral information of building materials through hyperspectral imaging and perform joint reasoning with the spatiotemporal feature vector set to obtain the defect feature set of building materials; Step 4: Input the building material defect feature set into the improved PinSAGE model, and perform spatial modeling through the feature map construction module, spatial local relation encoder, global dependency encoder and spatial dependency construction module to obtain the spatial dependency graph; Step 5: Perform defect reasoning on the spatial dependency graph to predict building material defects and construct a defect evolution prediction model; Step Six: Based on the aforementioned defect evolution prediction model, generate a defect detection strategy for building materials using a rule engine; Step 7: Locate the defects in the building materials according to the defect detection strategy and obtain a defect detection report.

[0007] Optionally, step one specifically includes: An RGB camera is used to capture the color, shape, and texture information of building material surfaces to obtain the two-dimensional visual characteristics of the building materials; Infrared thermal imagers are used to acquire thermal image data of building materials. By detecting the temperature distribution and changes on the surface of building materials, the infrared thermal imagers can identify potential internal defects, including cracks, voids, and corrosion. The structured light sensor is used to acquire depth information of building materials. The structured light sensor constructs three-dimensional surface data by projecting light spots and analyzing reflections. The raw multimodal data is preprocessed, including the two-dimensional visual features of building materials, thermal image data, and depth information. The preprocessing steps include outlier removal, missing value imputation, spatial coordinate alignment, and normalization of different types of data in the original multimodal data to obtain a building materials dataset with a unified structure.

[0008] Optionally, the dual-stream spatiotemporal mechanism specifically refers to: The building material dataset is processed by convolutional neural networks using spatial flow to extract texture, shape, and edge features of the building material surface; The convolutional processing involves extracting local features from the image through multiple convolutional layers, constructing a visual feature representation layer by layer, and obtaining the spatial structural feature vector of the building materials. The building material dataset is modeled temporally in chronological order using gated recursive units based on time flow. The dynamic evolution of building material defects is captured through the recursive structure, and the trend of defect changes over time is analyzed to obtain time series feature vectors. By combining the spatial structure feature vector and the time series feature vector, a spatiotemporal feature vector set of building materials is obtained.

[0009] Optionally, step three specifically includes: Hyperspectral imaging is used to acquire spectral data of building material surfaces. The hyperspectral imaging technology captures spectral changes of building material surfaces by acquiring reflectance spectra at different wavelengths. By jointly reasoning with the spectral data and the spatiotemporal feature vector set, a feature set of building material defects is obtained. This joint reasoning includes: Gaussian filtering is used to denoise the spectral data acquired by hyperspectral imaging, normalizing the spectral data of each wavelength to a uniform numerical range, eliminating the scale difference of spectral values ​​of different wavelengths, and obtaining standardized spectral data. A joint feature vector is obtained by concatenating standardized spectral data with spatiotemporal feature vectors. The concatenation operation is a merging operation along the feature dimension. The spectral data has multiple channels with multiple wavelengths, and the spatiotemporal feature vector contains spatial features and time series features. The joint feature vector includes the spatial structure, temporal evolution information, and spectral properties of building materials. By setting a threshold to remove feature dimensions in the joint feature vector that are less than a preset threshold and are irrelevant to defects, a feature set of building material defects is obtained.

[0010] Optionally, the improved PinSAGE model is specifically as follows: The feature set of building material defects is input into the feature map construction module. The feature map construction module divides the building material into multiple building material regions of equal size and assigns each building material region to a building material defect feature to obtain a node feature vector. Each building material region is treated as a node, and the edges between nodes represent the spatial distance between the corresponding building material regions, thus obtaining a preliminary feature map of the building materials. The preliminary feature map is input into the spatial local relation encoder, which captures local structural information by extracting the spatial adjacency relationship between nodes; Calculate the Euclidean distance between the node feature vectors corresponding to each pair of adjacent nodes in the local structural information. If the Euclidean distance is less than a preset distance threshold, sum the corresponding node feature vectors to obtain a local relation vector. The local relation vector is used to describe the spatial dependency between defect regions. The local relation vector is input into the global dependency encoder, which establishes global dependencies by calculating the similarity between the local relation vectors, and obtains a global dependency vector, which represents the defect distribution trend of building materials; The similarity between local relation vectors is used to establish global dependencies: calculate the Manhattan distance between every two local relation vectors. If the Manhattan distance is less than a preset similarity threshold, then the corresponding local relation vectors are weighted and averaged. The global dependency vector is input into the spatial dependency construction module. The similarity matrix between building material regions is calculated based on the cosine similarity between the global dependency vectors. The similarity matrix represents the spatial relationship between the various regions of the building material. Each global dependency vector is used as a node and connected according to the similarity matrix. The weight of the edge is the cosine similarity between two corresponding building material regions in the similarity matrix, thus obtaining a spatial dependency graph.

[0011] Optionally, step five specifically includes: Defect reasoning is performed based on the spatial dependency graph, wherein the defect reasoning starts from the defect node and propagates using the spatial dependency relationship in the spatial dependency graph. By analyzing the propagation information between defective nodes and utilizing the topological structure of the spatial dependency graph, the expansion trend and mutual influence of each defective region are analyzed. Through multiple rounds of iterative reasoning, the potential defective regions and evolution trends in building materials are inferred. Based on the potential defect areas and evolution trends in the reasoning, the evolution trend of the defect areas is modeled as time series data. By modeling the changes of the defect areas at different time points, the expansion and evolution patterns of the defect areas are captured, resulting in a time-space coupled defect evolution model. Based on the aforementioned defect evolution model, regression analysis is performed on historical defect data to predict the changes in various defect areas of building materials over a future period. These changes include the expansion of cracks, the enlargement of cavities, and corrosion. By fusing the inference results and prediction results, a final defect evolution prediction model is constructed.

[0012] Optionally, step six specifically includes: The detection process of the defect evolution prediction model is adjusted according to environmental changes through a rule engine, which includes: When the lighting conditions in the detection environment change, the rule engine adjusts the contrast or brightness of the defects according to the light intensity and distribution. Based on the different types of building materials, the rule engine automatically selects convolution kernels of different sizes for convolution operations; Based on the predicted defect type, the rule engine selects different detection processes, which include local area analysis for cracks and global detection strategy for large-area corrosion. The rule engine generates a defect detection strategy, which automatically adjusts the detection method under different environments, lighting conditions, material types, and defect types.

[0013] Optionally, step seven specifically includes: According to the defect detection strategy, by analyzing building material data, defects in building materials are accurately located, the position of defects on the surface of building materials is marked, and the position coordinates and shape information of the defect area are generated. The system identifies the type of defect in the located defect area, determines the specific type of defect, and confirms and classifies the defect type by analyzing the shape, size, color, and texture information of the defect area, combined with historical data and defect evolution trends. Generate a defect detection report for each defect based on the defect type.

[0014] The beneficial effects of this invention are: This invention overcomes the limitations of existing technologies in building material defect detection by combining multimodal sensor data, spatiotemporal feature extraction, hyperspectral imaging technology, and spatial modeling. Traditional defect detection methods typically rely on a single data source and static images, failing to effectively identify microcracks, deep defects, and their evolution trends in complex backgrounds. This invention, however, acquires multimodal data such as RGB images, infrared thermograms, and structured light depth information from multiple sensors, comprehensively reflecting the surface structure and internal state of building materials, providing richer input information for subsequent defect detection. In particular, the introduction of hyperspectral imaging technology captures spectral changes in building materials across different wavelength ranges, effectively detecting microcracks and deep defects that traditional methods cannot identify, improving detection accuracy and comprehensiveness. Through feature extraction based on a dual-stream spatiotemporal mechanism, this invention not only extracts spatial features but also captures the evolution of defects over time, solving the problem of traditional methods' inability to dynamically track defect expansion and changes. Defect evolution is typically a slow and gradual process, especially the expansion of cracks and the intensification of corrosion, which traditional methods struggle to anticipate. This invention, through a defect evolution prediction model, can predict the future development trend of defects, thereby providing forward-looking data support for the maintenance and repair of buildings, identifying potential safety hazards in advance, and preventing further deterioration of structural damage.

[0015] Furthermore, by combining the improved PinSAGE model with spatial modeling technology, this invention can model the spatial relationships of building material defects using graph neural networks, accurately identifying the interdependencies between defective areas, thus making defect detection more comprehensive and accurate. During defect location and repair suggestion generation, this invention can precisely pinpoint defect locations and provide specific repair methods, materials, and technical solutions based on defect types, providing a scientific basis for subsequent repair work. The combination of these technologies enables the building material defect detection system of this invention to possess high-precision, comprehensive, and dynamic prediction capabilities, significantly improving the efficiency and accuracy of building material quality testing, while providing reliable support for the long-term maintenance and safety of buildings. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0017] Figure 1 This is an overall flowchart of a visual inspection-based intelligent identification method for building material defects proposed in this invention. Figure 2 This is a schematic diagram illustrating the steps of generating a feature set of building material defects in a visual detection-based intelligent identification method for building material defects proposed in this invention. Figure 3 This is a flowchart of the improved PinSAGE model processing steps for an intelligent identification method for building material defects based on visual inspection proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figure 1-3 A method for intelligent identification of defects in building materials based on visual inspection includes the following steps: Step 1: Collect raw multimodal data of building materials through multiple sensors, and preprocess the raw multimodal data to generate a building material dataset; Step 2: Based on the dual-stream spatiotemporal mechanism, spatial and temporal features are extracted from the building material dataset to obtain a spatiotemporal feature vector set; Step 3: Obtain the spectral information of building materials through hyperspectral imaging and perform joint reasoning with the spatiotemporal feature vector set to obtain the defect feature set of building materials; Step 4: Input the building material defect feature set into the improved PinSAGE model, and perform spatial modeling through the feature map construction module, spatial local relation encoder, global dependency encoder and spatial dependency construction module to obtain the spatial dependency graph; Step 5: Perform defect reasoning on the spatial dependency graph to predict building material defects and construct a defect evolution prediction model; Step Six: Based on the aforementioned defect evolution prediction model, generate a defect detection strategy for building materials using a rule engine; Step 7: Locate the defects in the building materials according to the defect detection strategy and obtain a defect detection report.

[0020] In this embodiment, step one specifically includes: An RGB camera is used to capture the color, shape, and texture information of building material surfaces, acquiring their two-dimensional visual characteristics. The RGB camera can capture surface details and generate high-resolution images, providing a detailed description of the material's surface features. By analyzing the texture, color, and shape of the building material surface, the system can identify potential surface defects, such as cracks, damage, or surface aging, which are often difficult to detect directly with the naked eye in their early stages. RGB image data provides crucial foundational information for subsequent defect identification and classification.

[0021] Infrared thermal imagers are used to acquire thermal image data of building materials. By sensing the temperature distribution and changes on the surface of building materials, infrared thermal imagers help detect potential defects that cannot be identified by conventional visual inspection methods, such as heat loss, internal cracks, corrosion, and voids. Infrared thermal imagers can provide effective information based on differences in surface temperature, especially when there are hidden defects inside building materials. By capturing reflected images of heat changes, they can reveal potential problems in the building structure. These internal defects include, but are not limited to, cracks, voids, and corrosion, which may not be easily detected by visual inspection from the inside or deep layers of the material.

[0022] Structured light sensors are used to acquire depth information of building materials. By projecting light spots of a known pattern and analyzing their reflection, structured light sensors construct a three-dimensional data model of the building material surface. Structured light technology offers high accuracy in precisely measuring surface details, and is particularly suitable for building materials with irregular or complex shapes. Through processing the three-dimensional data, the system can accurately identify surface microcracks, dents, protrusions, or other irregularly shaped defects. The acquired depth information not only provides effective support for detecting surface defects but also assists in locating the specific position of defects using precise spatial data.

[0023] The raw multimodal data is preprocessed. This raw multimodal data includes two-dimensional visual features from an RGB camera, thermal image data from an infrared thermal imager, and depth information from a structured light sensor. After these different types of sensor data are processed uniformly, more comprehensive information on building material defects can be provided, offering more accurate data for subsequent analysis.

[0024] The preprocessing steps include outlier removal, missing value imputation, spatial coordinate alignment, and normalization for different types of data in the original multimodal data. Outlier removal removes invalid data caused by sensor errors or environmental interference, ensuring the reliability of subsequent analysis. Missing value imputation fills in missing data caused by insufficient data acquisition or sensor failure, avoiding the impact of empty data on subsequent processing. Spatial coordinate alignment ensures that data acquired by different sensors can be accurately aligned within the same spatial reference frame, ensuring data uniformity and consistency. Normalization standardizes data acquired by different sensors, making data from different sensors comparable and laying the foundation for subsequent fusion and feature extraction.

[0025] After these preprocessing steps, a unified building materials dataset is finally obtained, providing high-quality data input for subsequent feature extraction, defect identification, and spatial modeling.

[0026] In this embodiment, the dual-stream spatiotemporal mechanism is specifically as follows: The building material dataset is processed using a convolutional neural network (CNN) via spatial flow to extract texture, shape, and edge features from the building material surfaces. A CNN is a powerful image processing tool that progressively extracts local features from an image through convolution operations. Each convolutional layer processes the image through filters, extracting more abstract features to identify detailed information about the building material surface, such as cracks, dents, and variations in surface texture. These features play a crucial role in subsequent defect identification. The convolutional processing extracts local features from the image through multiple convolutional layers, constructing a visual feature representation layer by layer, from low-level edge features to high-level structural features, ultimately yielding a spatial structural feature vector of the building material. This feature vector includes spatial structural information about the building material surface, providing foundational data for subsequent defect localization and analysis.

[0027] The system uses gated recurrent units (GRUs) to perform temporal series modeling on the building material dataset, analyzing the changing trends of building material defects over time. GRUs, as an advanced temporal modeling method, effectively capture long-term dependencies in time series. In building material defect detection, defect changes often exhibit temporal evolution characteristics, such as the gradual expansion of cracks and the intensification of corrosion. GRUs can capture these changes through recursive structures, analyzing the evolutionary trends of defects over time. Through this process, the system can generate time series feature vectors that reflect the temporal evolution characteristics of building material defects.

[0028] By combining the spatial structural feature vectors and the time-series feature vectors, a spatiotemporal feature vector set for building materials is obtained. Combining spatial and temporal characteristics allows for a comprehensive description of the defect state of building materials, considering not only their spatial distribution but also their dynamic changes. This spatiotemporal feature vector set provides multi-dimensional data support for subsequent defect reasoning, evolution prediction, and repair suggestion generation, enabling the system to accurately identify and predict defects in building materials in more complex environments. This combination of a dual-stream spatiotemporal mechanism significantly improves the accuracy and robustness of defect identification, especially when facing defects of different types and at different time stages, allowing for better analysis and early warning.

[0029] In this embodiment, step three specifically includes: Hyperspectral imaging is used to acquire spectral data of building material surfaces. By acquiring reflectance spectra at different wavelengths, hyperspectral imaging technology can capture detailed spectral variations on building material surfaces. The core advantage of this technology lies in its high-resolution spectral information, providing richer information than traditional image processing methods. Particularly in the detection of potential internal defects in materials, hyperspectral imaging can reveal problems such as microcracks, corrosion, or moisture penetration, defects that are often difficult to detect using traditional RGB images or infrared imaging. By acquiring reflectance spectra at different wavelengths, hyperspectral imaging can accurately identify physical changes and internal defects in materials across the spectral domain.

[0030] By jointly reasoning with the spectral data and the spatiotemporal feature vector set, a defect feature set for building materials is obtained. In this process, the spectral data and the spatiotemporal feature vector set are integrated to leverage their complementary advantages and enhance the accuracy and comprehensiveness of defect detection. Specifically, the spectral data provides information on the physical properties of building materials, while the spatiotemporal feature vector reflects the spatial distribution and temporal evolution characteristics of defects. Joint reasoning, by integrating these two types of information, enables more accurate location of defects in building materials and allows for multi-dimensional analysis.

[0031] The joint reasoning includes the following steps: Gaussian filtering was used to denoise the spectral data acquired by hyperspectral imaging. Since spectral imaging data is often affected by noise, Gaussian filtering can effectively remove high-frequency noise from the image, maintain signal smoothness, and thus improve data usability and the accuracy of subsequent analysis.

[0032] Normalization normalizes the spectral data for each wavelength to a uniform numerical range, eliminating scale differences in spectral values ​​across different wavelengths. Since the reflectance spectral values ​​for each wavelength in hyperspectral imaging may have different numerical ranges and distributions, normalization ensures that the spectral data for all wavelengths are within the same numerical range, thereby avoiding errors caused by scale differences and ensuring data consistency.

[0033] Standardized spectral data is concatenated with spatiotemporal feature vectors to obtain a joint feature vector. This concatenation operation merges the spectral data and spatiotemporal feature vectors along the feature dimension, forming a comprehensive feature representation. Spectral data contains multiple channels across multiple wavelengths, while the spatiotemporal feature vectors include the spatial and temporal characteristics of building materials. Through concatenation, the resulting joint feature vector can simultaneously express the spatial structure, temporal evolution information, and spectral properties of building materials, thus providing richer feature support for accurate defect identification.

[0034] By setting a threshold, feature dimensions that are less than a preset threshold and therefore irrelevant to defects are removed from the joint feature vector. To improve the effectiveness of the feature vector, it is necessary to remove feature dimensions that are irrelevant to defects and may introduce noise. By setting a reasonable threshold, the system can filter out unimportant features and retain information meaningful for defect identification, thereby enhancing the accuracy and efficiency of defect detection.

[0035] Finally, the joint feature vector of the spectral data and spatiotemporal features processed through the above steps generates a defect feature set for building materials. These defect feature sets can be used for subsequent tasks such as defect reasoning, evolution prediction, and repair suggestion generation, ensuring comprehensive, accurate, and efficient identification of building material defects.

[0036] Through the above steps, this implementation method can effectively integrate spectral information and spatiotemporal characteristics to improve the accuracy of building material defect detection, especially when facing complex, minute or deep defects, providing more reliable detection results.

[0037] In this embodiment, the improved PinSAGE model is specifically as follows: The feature set of building material defects is input into the feature map construction module. The feature map construction module divides the building material into multiple building material regions of equal size and assigns each building material region to a building material defect feature to obtain a node feature vector. Each building material region is treated as a node, and the edges between nodes represent the spatial distance between the corresponding building material regions, thus obtaining a preliminary feature map of the building materials. The preliminary feature map is input into the spatial local relation encoder, which captures local structural information by extracting the spatial adjacency relationship between nodes; Calculate the Euclidean distance between the node feature vectors corresponding to each pair of adjacent nodes in the local structural information. If the Euclidean distance is less than a preset distance threshold, sum the corresponding node feature vectors to obtain a local relation vector. The local relation vector is used to describe the spatial dependency between defect regions. The local relation vector is input into the global dependency encoder, which establishes global dependencies by calculating the similarity between the local relation vectors, and obtains a global dependency vector, which represents the defect distribution trend of building materials; The similarity between local relation vectors is used to establish global dependencies: calculate the Manhattan distance between every two local relation vectors. If the Manhattan distance is less than a preset similarity threshold, then the corresponding local relation vectors are weighted and averaged. The global dependency vector is input into the spatial dependency construction module. The similarity matrix between building material regions is calculated based on the cosine similarity between the global dependency vectors. The similarity matrix represents the spatial relationship between the various regions of the building material. Each global dependency vector is used as a node and connected according to the similarity matrix. The weight of the edge is the cosine similarity between two corresponding building material regions in the similarity matrix, thus obtaining a spatial dependency graph.

[0038] The improved PinSAGE model proposed in this invention effectively enhances the accuracy and efficiency of building material defect identification by introducing a feature map construction module, a spatial local relation encoder, and a global dependency encoder. The feature map construction module meticulously divides the building material and generates node feature vectors, enabling the model to accurately describe the characteristics of each region of the material and capture the spatial dependencies between different regions. The spatial local relation encoder accurately captures the local structural information of the building material surface by analyzing the Euclidean distance between adjacent nodes, which helps identify minute cracks and irregular defects. The global dependency encoder establishes the defect distribution trend of the building material by calculating the similarity of local relation vectors, further enhancing the model's ability to identify defects in large-scale areas. Combined with the spatial dependency graph of the global dependency, the model can comprehensively consider the global and local features of the building material, providing more comprehensive and accurate defect detection results. The application of this improved model significantly improves the identification ability of complex materials and various defect types, especially performing exceptionally well in the detection of subtle and deep defects.

[0039] In this embodiment, step five specifically includes: Defect reasoning is performed based on the aforementioned spatial dependency graph. This reasoning starts from defect nodes and propagates through the spatial dependencies within the graph. The spatial dependency graph is a graph structure obtained by modeling the spatial relationships between different regions on the surface of a building material. In this graph, nodes represent different regions of the building material, and edges represent the spatial dependencies between these regions. Defect nodes are nodes in the graph that represent defective areas of the building material. These nodes are connected by edges in the spatial dependency graph, indicating the spatial mutual influence between defects. The reasoning process begins from these defect nodes and propagates defect information to other adjacent regions by utilizing the dependencies in the graph, thereby capturing potential defective areas within the building material.

[0040] By leveraging the propagation information between defect nodes, the expansion trends and mutual influences of each defect region are analyzed using the topological structure of a spatial dependency graph. The topological structure of the spatial dependency graph represents the connections between nodes, reflecting the spatial relationships between different areas of the building material surface. In this reasoning process, the system can calculate the expansion direction, speed, and mutual influence of defects based on the adjacent nodes and connecting edges of each defect region through the propagation process. Through multiple rounds of iterative reasoning, the system can gradually infer potential defect regions in the building material that have not yet manifested and their evolution trends. This process effectively captures the dynamic changes of defects in building materials, especially those defects that gradually develop and expand due to time or environmental factors.

[0041] Based on the inferred potential defect areas and their evolution trends, the evolution trends of these defect areas are modeled as time-series data. The evolution of defect areas involves not only spatial expansion but also temporal changes. Therefore, the system models the changes in defect areas at different points in time to capture the expansion and evolution patterns of defect areas over time. At this point, the change information of the defect areas is transformed into time-series data. Combining spatial and temporal information, a defect evolution model that can simultaneously describe spatial distribution and temporal evolution is obtained. This model can reflect the full picture of building material defects, including crack propagation and corrosion aggravation, thus providing a reliable basis for further analysis and early warning.

[0042] Based on the aforementioned defect evolution model, regression analysis of historical defect data is used to predict the changes in various defect areas of building materials over future periods. Historical defect data includes defect information recorded at different points in time. Through regression analysis, the system can fit the expansion trend of defects based on historical data and predict their evolution over a future period. These changes include the expansion of cracks, the enlargement of cavities, and the intensification of corrosion. The regression model based on historical data is used to derive potential future defect changes, thus identifying potential building safety hazards in advance.

[0043] By fusing the inference and prediction results, a final defect evolution prediction model is constructed. This model integrates the defect region expansion trend obtained from spatial dependency graph inference and the future evolution prediction obtained from regression analysis, forming a comprehensive defect evolution prediction model. This final defect evolution prediction model can not only accurately predict the future development of current defects in building materials, but also identify potential defect areas that have not yet manifested. This model provides a scientific basis for building maintenance, repair, and preventative reinforcement, improving the foresight and reliability of building material testing.

[0044] Through the above steps, this implementation method can effectively predict future changes in building material defects, provide early warning and decision support for building safety management, and ensure the long-term stability and safety of buildings.

[0045] In this embodiment, step six specifically includes: The rule engine adjusts the detection process of the defect evolution prediction model according to environmental changes. This rule engine automatically and dynamically adjusts the defect detection strategy for building materials based on real-time environmental changes, optimizing detection results. The rule engine automatically adjusts and selects appropriate parameters and algorithms based on preset rules and real-time input information to adapt to different detection environments and conditions.

[0046] When lighting conditions change in the detection environment, the rule engine automatically adjusts the contrast or brightness of the image based on the light intensity and distribution. In the defect detection of building materials, changes in lighting often affect image quality, especially under low or high light conditions, where defects such as cracks and corrosion in building materials may become difficult to detect. To address this challenge, the rule engine automatically increases the contrast or brightness of defect areas based on the intensity and distribution of light, ensuring that defects in building materials are clearly presented under any lighting conditions and avoiding missed defects caused by changes in lighting.

[0047] Depending on the type of building material, the rule engine automatically selects convolution kernels of different sizes for convolution operations. Building materials vary greatly in type and surface characteristics; for example, concrete, metal, and wood differ significantly in surface texture, color, and shape. The rule engine can recognize the characteristics of different materials and automatically adjust the size of the convolution kernels in the convolutional neural network (CNN) to adapt to the extraction of surface features from different materials. For example, for metal surfaces with complex textures, using a smaller convolution kernel can extract fine surface defects; while for materials with smoother surfaces, using a larger convolution kernel helps capture a wider range of defect features.

[0048] Based on the predicted defect type, the rule engine selects different detection processes. Building material defects typically manifest in different ways, such as cracks, voids, and corrosion, and each defect type requires a different detection method. The rule engine automatically selects the most suitable detection process based on the defect type prediction results. For cracks, the rule engine chooses a local area analysis method, improving the accuracy of crack detection through refined local area scanning and processing; while for defects such as large-area corrosion, the rule engine chooses a global detection strategy, better capturing the overall situation of the corroded area through global image processing and analysis.

[0049] The rule engine generates a defect detection strategy that automatically adjusts the detection method under different environments, lighting conditions, material types, and defect types. This strategy not only considers environmental factors and material types but also automatically selects the optimal image processing algorithm, feature extraction method, and defect detection technology based on the nature of different defects. This adaptive strategy ensures that the defect detection method maintains high efficiency and accuracy in various practical application environments. Through flexible adjustment and optimization, the system can minimize false positives and false negatives, improving the overall accuracy and reliability of building material defect identification.

[0050] Through these adjustments, the rule engine can dynamically adapt to various complex environments and material properties, providing strong adaptability for defect detection and significantly improving the system's performance in practical applications.

[0051] In this embodiment, step seven specifically includes: In this embodiment, step seven specifically includes: According to the aforementioned defect detection strategy, defects in building materials are precisely located by analyzing the material data. The system identifies the defects' positions on the material surface and generates the location coordinates and shape information of the defect areas. In this process, defect localization relies not only on the original image data but also on previously extracted multimodal data, including RGB images, infrared thermal images, and depth images. Through precise image processing algorithms and spatial relationship modeling, the system can accurately pinpoint the location of defects on the building material surface and obtain the specific coordinates and morphological characteristics of the defect areas. This information is crucial for subsequent defect analysis and repair plan development, ensuring that the location and shape of defects are accurately recorded and analyzed.

[0052] The system identifies the type of defect in the located defect area, determining its specific type by analyzing its morphology, size, color, and texture, combined with historical data and defect evolution trends. In this step, the system not only performs a preliminary classification based on the current defect area morphology (e.g., cracks, voids, corrosion) but also analyzes the further development of the defect by incorporating historical defect data and evolution trends. Through comprehensive analysis of color changes, texture features, and size, the system accurately identifies the defect type, avoiding misclassification based on a single feature. The accuracy of this identification process is crucial, especially in environments with complex materials or multiple defects, effectively preventing misjudgments and omissions.

[0053] Based on the defect type, a defect detection report is generated for each type of defect. When generating the report, the system creates detailed content based on the defect's type, location, shape, size, and development trend. This report includes not only the defect's specific location coordinates, shape information, and type classification, but also repair recommendations. For different types of defects, the report details recommended repair materials, methods, and technical solutions, and proposes a priority repair order based on the defect's severity and potential impact. Furthermore, the report includes historical data comparisons and defect evolution trend analysis to help engineers understand the defect formation process and provide a basis for the rational planning of repair work.

[0054] Through the above steps, the system can accurately locate, identify, and classify defects in building materials, and generate detailed and comprehensive defect detection reports, providing a scientific basis for subsequent repair work and long-term maintenance of buildings.

[0055] Example 1: To verify the feasibility of this invention in practice, it was applied to the detection of surface defects in concrete of an industrial building. Located in an urban industrial area, this building is exposed to a harsh environment for extended periods, resulting in various types of surface defects, including cracks, corrosion, and voids caused by moisture. Since these defects can severely impact the stability and safety of the building, timely and accurate detection and location of these defects, especially in the early stages of micro-cracks and deep corrosion, is crucial. However, traditional manual inspection methods often rely on periodic checks and are ineffective at detecting small cracks or deep corrosion areas, easily leading to missed or false detections and low efficiency.

[0056] In this scenario, the visual inspection-based intelligent identification method for building material defects provided by this invention utilizes multiple sensors working collaboratively, combining an RGB camera, an infrared thermal imager, and a structured light sensor to comprehensively collect data from the surface of the building material. These sensors are used to collect different types of information: the RGB camera captures the color, shape, and texture of the concrete surface; the infrared thermal imager records the temperature distribution, helping to identify potential internal cracks and corrosion; and the structured light sensor provides precise three-dimensional depth information, further aiding in the identification of micro-cracks and surface irregularities. After preprocessing, the collected data is transformed into a unified building material dataset containing RGB images, thermal image data, and depth information. This data undergoes feature extraction using a dual-stream spatiotemporal mechanism, capturing defect features of the building material from both spatial and temporal dimensions. The spatial stream extracts the texture, shape, and edge features of the building material surface using a convolutional neural network; while the temporal stream uses a gated recurrent unit (GRU) model to perform temporal modeling of the data, capturing the evolution of defects. Combining these spatiotemporal features, the system can accurately identify micro-cracks and potential corrosion areas on the concrete surface.

[0057] Hyperspectral imaging technology was further used to capture spectral data of the building material surface. Through high-precision spectral reflectance information, hyperspectral imaging can reveal deep cracks and corrosion areas that traditional infrared thermal imagers cannot identify. This spectral data, after denoising, standardization, and concatenation with spatiotemporal feature vectors, yields a high-precision feature set of building material defects, enhancing the comprehensiveness and accuracy of defect identification. Through joint inference, the system can effectively fuse multimodal data from different sensors, providing multi-dimensional support for defect localization and analysis. The improved PinSAGE model performs spatial modeling in the spatial dependency graph, further enhancing the model's defect identification capability. This model progressively extracts and fuses defect information from the building material surface through modules such as feature map construction, spatial local relation encoder, and global dependency encoder, ultimately constructing a spatial dependency graph. In this process, the model not only considers the local relationships of different regions on the building material surface but also further enhances the comprehensive defect capture capability through global dependencies. Through the spatial dependency graph, the model can effectively identify the mutual influence and expansion trends between defect regions.

[0058] Based on the spatial dependency graph, the system performs defect inference and constructs a defect evolution prediction model. This model uses regression analysis of historical defect data to predict the changes in various defect areas within building materials over future time periods. For example, the system predicts that cracks may expand over time, and the area of ​​corrosion may gradually increase under moisture conditions. This predictive capability provides forward-looking guidance for building maintenance and repair. To further optimize the detection process, the rule engine automatically adjusts the detection strategy according to changes in the site environment. When the lighting conditions of the detection environment change, the rule engine automatically increases the contrast or brightness of the defect area to ensure accurate identification of defects under various lighting conditions. Furthermore, the rule engine automatically selects appropriate convolution kernel sizes for convolution operations based on different building material types to improve the accuracy of feature extraction. For cracks, the rule engine selects a local area analysis method, while for large-area corrosion, a global detection strategy is adopted. Through these intelligent adjustments, the defect detection strategy can dynamically adapt to the site environment, ensuring efficient and accurate identification of defects in building materials. The system accurately locates defects in building materials based on the defect detection strategy and generates detailed defect detection reports. The report includes the specific location, type, and remediation recommendations for each defect, providing a scientific basis for subsequent remediation work. The detection performance of traditional methods and the method of this invention was compared under the same detection scenario; the specific data are as follows:

[0059] Table 1 Comparative Analysis of Defect Detection Methods for Building Materials

[0060] Analysis of the data in Table 1 reveals that visual inspection methods typically rely on manual visual inspection or simple image processing algorithms, suitable for obvious surface defects. However, due to their strong subjectivity, they are prone to missing microcracks and deep defects, resulting in low efficiency and limitations imposed by the inspector's experience. In complex environments, accuracy and efficiency are difficult to guarantee. Infrared thermography can effectively detect internal defects in building materials, especially cracks and corrosion, but its ability to identify surface textures and microcracks is limited. Because infrared thermal imagers are sensitive to temperature changes, they may be affected by external temperature fluctuations in complex environments, leading to unstable detection results. Ultrasonic testing is mainly used to detect deep defects, identifying defect areas by analyzing reflected signals through sound wave propagation. While it is good at identifying deep defects, it has limitations in detecting surface defects (such as microcracks). Furthermore, this method is complex to operate, requires specialized personnel, is costly, and cannot quickly process large areas of material surfaces. The method of this invention combines multimodal sensors (RGB cameras, infrared thermal imagers, structured light sensors, etc.) and advanced spatiotemporal feature extraction technology, making defect detection in building materials more accurate and efficient. By combining hyperspectral imaging and spatiotemporal feature vector inference, this method can accurately identify minute cracks and deep defects. Furthermore, its high degree of automation significantly reduces manual intervention, resulting in significantly improved detection accuracy and efficiency, making it particularly suitable for building material inspection in large-scale and complex environments. Compared to traditional methods, this invention provides a more accurate and comprehensive defect detection solution for building materials, capable of efficiently identifying multiple types of defects in a shorter time, and performing exceptionally well when dealing with complex or minute defects.

[0061] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for intelligent identification of defects in building materials based on visual inspection, characterized in that, Includes the following steps: Step 1: Collect raw multimodal data of building materials through multiple sensors, and preprocess the raw multimodal data to generate a building material dataset; Step 2: Based on the dual-stream spatiotemporal mechanism, spatial and temporal features are extracted from the building material dataset to obtain a spatiotemporal feature vector set; Step 3: Obtain the spectral information of building materials through hyperspectral imaging and perform joint reasoning with the spatiotemporal feature vector set to obtain the defect feature set of building materials; Step 4: Input the building material defect feature set into the improved PinSAGE model, and perform spatial modeling through the feature map construction module, spatial local relation encoder, global dependency encoder and spatial dependency construction module to obtain the spatial dependency graph; Step 5: Perform defect reasoning on the spatial dependency graph to predict building material defects and construct a defect evolution prediction model; Step Six: Based on the aforementioned defect evolution prediction model, generate a defect detection strategy for building materials using a rule engine; Step 7: Locate the defects in the building materials according to the defect detection strategy and obtain a defect detection report.

2. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, Step one specifically involves: An RGB camera is used to capture the color, shape, and texture information of building material surfaces to obtain the two-dimensional visual characteristics of the building materials; Infrared thermal imagers are used to acquire thermal image data of building materials. By detecting the temperature distribution and changes on the surface of building materials, the infrared thermal imagers can identify potential internal defects, including cracks, voids, and corrosion. The structured light sensor is used to acquire depth information of building materials. The structured light sensor constructs three-dimensional surface data by projecting light spots and analyzing reflections. The raw multimodal data is preprocessed, including the two-dimensional visual features of building materials, thermal image data, and depth information. The preprocessing steps include outlier removal, missing value imputation, spatial coordinate alignment, and normalization of different types of data in the original multimodal data to obtain a building materials dataset with a unified structure.

3. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, The dual-stream spatiotemporal mechanism is specifically as follows: The building material dataset is processed by convolutional neural networks using spatial flow to extract texture, shape, and edge features of the building material surface; The convolutional processing involves extracting local features from the image through multiple convolutional layers, constructing a visual feature representation layer by layer, and obtaining the spatial structural feature vector of the building materials. The building material dataset is modeled temporally in chronological order using gated recursive units based on time flow. The dynamic evolution of building material defects is captured through the recursive structure, and the trend of defect changes over time is analyzed to obtain time series feature vectors. By combining the spatial structure feature vector and the time series feature vector, a spatiotemporal feature vector set of building materials is obtained.

4. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, Step three specifically involves: Hyperspectral imaging is used to acquire spectral data of building material surfaces. The hyperspectral imaging technology captures spectral changes of building material surfaces by acquiring reflectance spectra at different wavelengths. By jointly reasoning with the spectral data and the spatiotemporal feature vector set, a feature set of building material defects is obtained. This joint reasoning includes: Gaussian filtering is used to denoise the spectral data acquired by hyperspectral imaging, normalizing the spectral data of each wavelength to a uniform numerical range, eliminating the scale difference of spectral values ​​of different wavelengths, and obtaining standardized spectral data. A joint feature vector is obtained by concatenating standardized spectral data with spatiotemporal feature vectors. The concatenation operation is a merging operation along the feature dimension. The spectral data has multiple channels with multiple wavelengths, and the spatiotemporal feature vector contains spatial features and time series features. The joint feature vector includes the spatial structure, temporal evolution information, and spectral properties of building materials. By setting a threshold to remove feature dimensions in the joint feature vector that are less than a preset threshold and are irrelevant to defects, a feature set of building material defects is obtained.

5. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, The improved PinSAGE model is specifically as follows: The feature set of building material defects is input into the feature map construction module. The feature map construction module divides the building material into multiple building material regions of equal size and assigns each building material region to a building material defect feature to obtain a node feature vector. Each building material region is treated as a node, and the edges between nodes represent the spatial distance between the corresponding building material regions, thus obtaining a preliminary feature map of the building materials. The preliminary feature map is input into the spatial local relation encoder, which captures local structural information by extracting the spatial adjacency relationship between nodes; Calculate the Euclidean distance between the node feature vectors corresponding to each pair of adjacent nodes in the local structural information. If the Euclidean distance is less than a preset distance threshold, sum the corresponding node feature vectors to obtain a local relation vector. The local relation vector is used to describe the spatial dependency between defect regions. The local relation vector is input into the global dependency encoder, which establishes global dependencies by calculating the similarity between the local relation vectors, and obtains a global dependency vector, which represents the defect distribution trend of building materials; The similarity between local relation vectors is used to establish global dependencies: calculate the Manhattan distance between every two local relation vectors. If the Manhattan distance is less than a preset similarity threshold, then the corresponding local relation vectors are weighted and averaged. The global dependency vector is input into the spatial dependency construction module. The similarity matrix between building material regions is calculated based on the cosine similarity between the global dependency vectors. The similarity matrix represents the spatial relationship between the various regions of the building material. Each global dependency vector is used as a node and connected according to the similarity matrix. The weight of the edge is the cosine similarity between two corresponding building material regions in the similarity matrix, thus obtaining a spatial dependency graph.

6. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, Step five specifically involves: Defect reasoning is performed based on the spatial dependency graph, wherein the defect reasoning starts from the defect node and propagates using the spatial dependency relationship in the spatial dependency graph. By analyzing the propagation information between defective nodes and utilizing the topological structure of the spatial dependency graph, the expansion trend and mutual influence of each defective region are analyzed. Through multiple rounds of iterative reasoning, the potential defective regions and evolution trends in building materials are inferred. Based on the potential defect areas and evolution trends in the reasoning, the evolution trend of the defect areas is modeled as time series data. By modeling the changes of the defect areas at different time points, the expansion and evolution patterns of the defect areas are captured, resulting in a time-space coupled defect evolution model. Based on the aforementioned defect evolution model, regression analysis is performed on historical defect data to predict the changes in various defect areas of building materials over a future period. These changes include the expansion of cracks, the enlargement of cavities, and corrosion. By fusing the inference results and prediction results, a final defect evolution prediction model is constructed.

7. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, Step six specifically involves: The detection process of the defect evolution prediction model is adjusted according to environmental changes through a rule engine, which includes: When the lighting conditions in the detection environment change, the rule engine adjusts the contrast or brightness of the defects according to the light intensity and distribution. Based on the different types of building materials, the rule engine automatically selects convolution kernels of different sizes for convolution operations; Based on the predicted defect type, the rule engine selects different detection processes, which include local area analysis for cracks and global detection strategy for large-area corrosion. The rule engine generates a defect detection strategy, which automatically adjusts the detection method under different environments, lighting conditions, material types, and defect types.

8. The intelligent identification method for building material defects based on visual inspection according to claim 1, characterized in that, Step seven specifically involves: According to the defect detection strategy, by analyzing building material data, defects in building materials are accurately located, the position of defects on the surface of building materials is marked, and the position coordinates and shape information of the defect area are generated. The system identifies the type of defect in the located defect area, determines the specific type of defect, and confirms and classifies the defect type by analyzing the shape, size, color, and texture information of the defect area, combined with historical data and defect evolution trends. Generate a defect detection report for each defect based on the defect type.

Images