A method and related equipment for intelligent detection of defects in building curtain walls

Abstract

This application provides an intelligent detection method and related equipment for building curtain wall defects, belonging to the field of computer vision technology. The method includes: acquiring building curtain wall images using a drone equipped with a high-resolution camera; annotating and preprocessing the acquired images, cropping them into smaller images using a fixed cropping window to generate positive and negative samples, thus expanding the dataset; constructing a defect detection model based on a Mask R-CNN network, embedding a coordinate attention module in its backbone network to enhance spatial feature perception, and introducing a Focal Loss function in the region proposal network to alleviate sample imbalance; in the inference stage, using a sliding window and multi-size pyramids for image segmentation prediction, and using connectivity testing to fuse the prediction results, ultimately achieving the localization and labeling of curtain wall defects. This invention effectively solves the problems of high-resolution image processing, multi-scale defect detection, and high sample annotation costs, significantly improving the automation and accuracy of curtain wall defect detection.

Description

Technical Field

[0001] This application relates to the fields of computer vision and building inspection technology, and in particular to an intelligent detection method and related equipment for building curtain wall defects. Background Technology

[0002] Building curtain walls, especially glass curtain walls, have become the mainstream exterior envelope structure for modern high-rise buildings. However, during long-term use, curtain walls are prone to defects such as glass breakage, stone cracks, and sealant peeling due to material aging, stress changes, and environmental influences. These defects not only affect aesthetics but also pose significant safety hazards.

[0003] Currently, the mainstream inspection method still relies on manual inspection, which has problems such as low efficiency, high cost, strong subjectivity and high risk of high-altitude operation, making it difficult to achieve rapid and full-coverage inspection of large-scale curtain walls.

[0004] With technological advancements, the combination of drones and machine vision has provided a new direction for curtain wall inspection. Drones can efficiently and safely acquire high-resolution images of high-altitude curtain walls. However, automatically and accurately identifying defects from these massive amounts of images still faces significant challenges: 1) The conflict between image size and network receptive field: Images taken by drones are usually extremely high resolution (e.g., 4000×3000 pixels), while deep learning models, due to hardware limitations, typically accept small input sizes (e.g., 640×640 pixels). Directly scaling the original image will lead to the loss of features of small defects (e.g., fine cracks).

[0005] 2) Challenges in multi-scale defect detection: Curtain wall defects vary greatly in size, ranging from broken glass several meters wide to cracks a few millimeters wide. Single-scale detection models are insufficient to cover all defects.

[0006] 3) Imbalanced samples and high annotation costs: Defects account for a very small percentage of the overall curtain wall, resulting in a severe imbalance between positive and negative samples. At the same time, pixel-level annotation of high-resolution images requires a lot of manpower and resources.

[0007] 4) Complex background interference: The curtain wall structure is complex, including window frames, decorative strips, etc., and its defects are easily drowned out by background noise.

[0008] In existing technologies, although some technical solutions attempt to use deep learning for curtain wall detection, most of them simply apply existing target detection models (such as YOLO and Faster R-CNN), failing to systematically solve the core engineering problems mentioned above, such as high-resolution images, multi-scale defects, sample generation and annotation, resulting in limited detection accuracy, generalization ability and practicality in actual combat. Summary of the Invention

[0009] The main objective of this application is to propose an intelligent detection method and related equipment for building curtain wall defects based on UAVs and deep learning. This method can effectively process high-resolution images, achieve accurate and efficient automatic identification of multi-scale defects, and significantly reduce data annotation costs.

[0010] To achieve the above objectives, one aspect of this application proposes an intelligent detection method for building curtain wall defects, the method comprising: Image acquisition: RGB images of the building's curtain wall are acquired using a high-resolution camera mounted on a drone; Data preprocessing: The acquired high-resolution image is annotated at the pixel level in one go; then, the data preprocessing is automatically performed by a script. The preprocessing includes: sliding cropping of the original annotated image based on a fixed-size cropping window to generate positive samples containing defective regions and negative samples without defective regions, and constructing a dataset with balanced positive and negative samples; at the same time, the original image is scaled at multiple resolutions to construct a sampling pyramid, and the positive and negative sample sampling is performed at each layer of the pyramid to achieve data augmentation and multi-scale dataset construction. Model building and training: An improved defect detection model is built based on the Mask R-CNN instance segmentation network; the constructed dataset is input into the improved defect detection model for training to obtain a trained model; Result prediction and output: The high-resolution image to be detected is segmented into sub-images through a sliding window and input into the trained model for prediction; the prediction results of the sub-images are mapped back to the coordinates of the original image, and the prediction results of adjacent masks belonging to the same defect category are fused based on connectivity analysis, and finally the defect category, location and mask information are output.

[0011] In some embodiments, the process of generating positive samples includes: using the center of the defect region as a reference, introducing a random offset to the center point of the clipping window to ensure that the clipping window contains at least part of the defect region; The process of generating negative samples includes: randomly selecting a background region in the original image that does not contain any defective areas and cropping it; For positive samples that contain only partial defect areas after cropping, polygon intersection operation is used to map the original annotations to the cropped sub-image; and an area threshold is set to filter out positive samples with too small defect areas after mapping.

[0012] In some embodiments, the positive sample sampling process further includes a sample filtering step: Calculate the area of ​​the defect region mapped to the cropping sub-image. If the area is less than a preset threshold, filter and discard the positive sample.

[0013] In some embodiments, at the top layer of the sampling pyramid, i.e. the layer with the smallest image size, when the image background area is smaller than the fixed-size cropping window, a padding method is used to ensure that the output sample size is consistent.

[0014] In some embodiments, the improvements to the improved defect detection model include: A coordinate attention module is embedded in the backbone network of Mask R-CNN; In the region proposal network of Mask R-CNN, Focal Loss is used as the classification loss function.

[0015] In some embodiments, the execution process of the coordinate attention module includes: Coordinate information embedding: Perform one-dimensional global pooling in the width and height directions on the input feature map to generate a pair of orientation-aware feature maps; Coordinate attention generation: After concatenating and transforming the orientation-aware feature map, it is divided into two independent tensors, and attention maps in the width and height directions are generated by convolution transformation respectively. The generated attention map is multiplied with the original input feature map to enhance the model's ability to perceive spatial location.

[0016] In some embodiments, the functional expression of Focal Loss is:

[0017] in, It is the model's predicted probability for positive and negative samples. γ is a coefficient used to balance class weights, and γ is a focusing parameter used to adjust the weights of easy and difficult samples.

[0018] In some embodiments, the step size of the sliding window is smaller than the window size to ensure that defects in the window edge area can be effectively detected; The connectivity analysis specifically involves determining whether the predicted defect masks in different subgraphs are spatially adjacent and belong to the same defect category. If so, the multiple prediction results are merged into a single defect target.

[0019] In some embodiments, during the prediction phase, a multi-resolution prediction pyramid is constructed with the same scaling factor as that used in the training phase. The image to be detected is scaled to each layer of the prediction pyramid, and the sliding window prediction and result mapping are performed independently at each layer; Finally, the prediction results of all layers of the pyramid are merged to form the final defect detection result.

[0020] In some embodiments, the building curtain wall defects include at least one of broken glass, stone cracks, and sealant detachment; The drone takes pictures in a matrix mosaic mode at a distance of 10 to 15 meters, and the resolution of the RGB image is not less than 4000×3000 pixels.

[0021] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0022] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0023] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0024] The embodiments of this application include at least the following beneficial effects: This application uses drones to collect high-resolution images of building curtain walls, which can avoid the dangers and inefficiencies of manually collecting curtain wall images. At the same time, clearer images can significantly improve the detection capability of small defects such as cracks. By preprocessing the original high-resolution images, the demand for computer hardware can be reduced, the efficiency of manual annotation can be improved, and the dataset size can be significantly increased under the premise of a limited number of original samples. At the same time, it can alleviate the problem of imbalance between positive and negative samples, which helps to train a neural network model with higher accuracy. In the complex background of building curtain walls, the improved neural network model has higher detection accuracy for various defects such as broken glass, cracked stone, and detached sealant, especially for defect types such as cracked stone that are small in size, have a small sample size, or are difficult to detect. The prediction post-processing and output can maintain the same resolution between the predicted image and the image during training, which helps to improve the detection accuracy, especially the detection accuracy of small-sized defects. The final output can intuitively display the location, type, and outline of the defect, which is more conducive to guiding the maintenance work of building curtain walls. It has good real-time performance and engineering application prospects. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the steps of an intelligent detection method for building curtain wall defects in an embodiment of this application; Figure 2 This is a schematic diagram of image acquisition and sample cropping in an embodiment of this application; Figure 3 This is a flowchart of the sampling algorithm in the embodiments of this application; Figure 4 This is a schematic diagram of multi-size defects in the embodiments of this application; Figure 5 This is a schematic diagram of a multi-scale sampling pyramid in an embodiment of this application; Figure 6 This is a schematic diagram of the cropping algorithm in an embodiment of this application; Figure 7 This is a diagram of the Mask R-CNN network structure in an embodiment of this application; Figure 8 This is a structural diagram of the coordinate attention module in an embodiment of this application; Figure 9 This is a flowchart of the sliding window reasoning and result fusion in an embodiment of this application; Figure 10 This is a schematic diagram of multi-scale prediction in an embodiment of this application; Figure 11 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] With urbanization and increasing building height, glass curtain walls, due to their lightweight and aesthetically pleasing design, have rapidly become the primary exterior envelope for modern high-rise buildings. However, due to early construction defects and long-term exposure to complex environments, curtain walls are prone to defects such as glass breakage and sealant detachment. This not only affects the building's aesthetics and functionality but may also threaten structural safety and human safety. Existing manual inspection methods suffer from low efficiency, high cost, and strong subjectivity, and are difficult to achieve large-area coverage at high altitudes. Therefore, exploring an intelligent and automated curtain wall inspection solution is an urgent problem to be solved.

[0029] The development of drone technology and machine vision defect detection technology has brought new possibilities for defect detection in building curtain walls. Compared with manual inspection, using drones to photograph building curtain walls has advantages such as high efficiency, high accuracy, and reliability. However, drone-captured images still require manual inspection of each image for defects. Combining drones and machine vision can significantly improve the efficiency and quality of curtain wall inspection. Machine vision detection algorithms can be divided into traditional algorithms and deep learning algorithms. Traditional image processing methods (such as Canny and SUSAN operators) can detect some defects in specific scenarios, but they require specialized design for specific objects. For example, traditional algorithms designed to detect broken glass cannot be used to detect defects such as detached sealing strips. When dealing with building curtain wall objects that simultaneously have multiple defect types such as broken glass, cracked stone, and detached sealing strips, with complex backgrounds and uncontrollable lighting conditions due to weather and shooting angle, traditional algorithms lack sufficient robustness and are difficult to meet the needs of actual engineering.

[0030] In recent years, deep learning-based detection methods have achieved good results in the field of building curtain wall and infrastructure inspection due to the powerful data fitting and inference prediction capabilities of deep learning. However, existing methods are still limited to the simple application of conventional multi-object detection frameworks and do not consider the challenges involved in building curtain wall detection based on UAVs and deep learning, such as the contradiction between large-size original images and limited receptive fields of neural networks, the large scale span of defects, the problem of efficient data annotation and automatic generation of multi-scale datasets, and the problem of efficient detection inference for high-resolution test images. For example, patent CN117808789A collects glass curtain wall cracks through a specific path, and after data augmentation, it is input into a convolutional neural network for training, which can obtain a glass curtain wall crack instance segmentation mask with higher accuracy than the YOLO network. However, this patent uses conventional data processing methods such as random flipping, which is suitable for detecting small-size input images after appropriate cropping, and cannot be directly applied to high-resolution images captured by UAVs. Patent CN115565068A uses a YOLO network for automatic detection of glass curtain wall damage and reduces the number of model parameters through sparse constraints and pruning operations, thus reducing the number of model parameters to some extent while maintaining accuracy. However, the basic YOLO network used in this patent, designed for multi-object detection, labels and predicts targets using bounding boxes of the target objects. It does not consider the characteristics of small defects such as cracks, which have a very low pixel ratio and diverse shapes within the bounding boxes. Therefore, it cannot guarantee the detection accuracy of such defects, nor can it output accurate location and shape information of the target objects. Furthermore, this method is only used for glass curtain wall detection, and its robustness to the detection of other types of curtain wall defects is still unclear.

[0031] In summary, combining drones with deep learning is an effective solution for building curtain wall inspection. However, in practical engineering applications, the following issues still need to be addressed: (1) The contradiction between large-size raw images and limited receptive fields of neural networks: Raw images captured by drones are large in size, with common resolutions such as 4000×3000. However, due to hardware limitations and time constraints, the image size used for neural network training is usually set much lower than this value (e.g., 640×640). Directly using raw-size images is basically infeasible due to the high requirements for computer hardware resources. How to process the raw images to a size suitable for network processing is a problem that needs to be solved, which further involves: a) Problem of loss of small-sized defect features: Defects in building curtain walls, such as stone cracks, are small in size in the direction of crack branch width. If the resolution of the original image is reduced (the size of the original image is reduced) to a size that the neural network can accept, the small-sized defect features such as cracks will be partially or completely lost, resulting in the missed detection of such defects.

[0032] b) Defect Sample Expansion and Labeling: Since defects occur relatively infrequently in real-world scenarios, collecting sufficient defect samples for building curtain walls is often difficult. A key challenge is generating a sufficient dataset of defect samples for training from limited original defect images while reducing image labeling costs. One easily implemented approach is to reduce the size of the original images, cropping them to a size acceptable to the neural network model, and then labeling each cropped image. However, this significantly increases labeling costs and reduces efficiency. Another approach is to directly label the original images and then transform the original label coordinates during the cropping and scaling process using an algorithm, which can reduce labeling costs. Therefore, a feasible data processing method is needed to address issues such as coordinate transformation and obtaining high-quality positive and negative samples. (2) Sample Data Balance Problem: To ensure the generalization ability of the network model, the data balance problem should be solved, that is, to ensure that there are enough sample instances of each type of defect in the training set. At the same time, for practical application, the detection of local defects should be considered, that is, when only a local area of ​​the defect is visible due to occlusion or exceeding the current field of view boundary, the local defect object should also be reliably detected. How to automatically generate various possible local defects during the dataset preparation process is another important aspect that the dataset generation algorithm needs to consider. If there are significant differences in the number of different types of defects in the actual samples, the detection neural network needs to be able to focus on samples with a smaller number of samples or those that are more difficult to predict, in the case of diverse defect types and an imbalance in the number of defect types, so as to improve the accuracy of all defect types as much as possible.

[0033] (3) Relative positional characteristics of various defect targets in the curtain wall background: The background of building curtain walls is complex and diverse. Defects account for a small percentage of the area in various curtain wall structures, but their locations often exhibit certain patterns. For example, glass is always surrounded by window frames, and colloids only exist within objects like window frames (not in the glass). Capturing the correlation between defects and their spatial location helps detect defects from complex backgrounds. Therefore, it is necessary to modify the existing general multi-target detection network structure to better utilize this characteristic and improve detection accuracy.

[0034] (4) Multi-scale detection problem: In addition to the training dataset, it is also necessary to solve the problem of how to directly detect defects of different scales from the original large-size test image. Similarly, the fundamental problem is that the receptive field of the network is too small relative to the test image. If the conventional sliding window detection method is used, it will not only be inefficient, but also misjudgment will occur because it cannot observe the overall features of the defect due to the "seeing only a part". For example, the size of broken glass is often an order of magnitude larger than the size of stone cracks, while the size of stone cracks and detached sealant is too small compared to the background. According to the data collected from the detection projects, in the 4000×3000 image taken by the UAV, the average size of broken glass is about 2283×2166, the average size of stone cracks is about 359×271, and the average size of detached sealant is about 186×410. Therefore, broken glass is suitable for prediction at lower resolution, while defects such as stone cracks are suitable for prediction at higher resolution. If multiple building curtain wall defects are to be trained and predicted simultaneously, it is necessary to solve the problem of how the network can accurately predict defects of different sizes from large images under limited hardware conditions when predicting images.

[0035] In view of this, this application provides an automatic detection method, electronic device, storage medium and program product for building curtain wall defects, which can achieve high-precision, high-efficiency and multi-category automatic identification of defects, greatly alleviate the problems of large image size, diverse sample types and shapes and small proportion of defects in building curtain wall detection, and is suitable for curtain wall safety monitoring tasks in complex scenarios.

[0036] This application provides an automatic detection method for building curtain wall defects, relating to the fields of image detection and computer vision technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle-mounted terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the automatic detection method for building curtain wall defects, but is not limited to the above forms.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] Figure 1 This is an optional flowchart of an automatic detection method for building curtain wall defects provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0039] Step S101, Image Acquisition: Acquire RGB images of the building curtain wall using a high-resolution camera mounted on a drone.

[0040] In some embodiments, a high-resolution camera mounted on a drone is used to fly along a specific path and with specific parameters (such as a distance of 10-15 meters from the curtain wall, in matrix mosaic mode) to capture high-resolution RGB images of the building curtain wall.

[0041] Step S102, data preprocessing: The acquired high-resolution image is annotated at the pixel level in one go; then the data preprocessing is automatically executed by a script. The preprocessing includes: sliding cropping of the original annotated image based on a fixed-size cropping window to generate positive samples containing defective regions and negative samples without defective regions, and constructing a dataset with balanced positive and negative samples; at the same time, the original image is scaled at multiple resolutions to construct a sampling pyramid, and the positive and negative sample sampling is performed at each layer of the pyramid to realize data augmentation and multi-scale dataset construction.

[0042] In some embodiments, the acquired high-resolution images undergo one-time pixel-level manual annotation. Subsequently, data preprocessing is performed using an automated script, including: 1) Positive and negative sample sampling: A fixed-size cropping window (e.g., 800×800 pixels) is used to perform sliding cropping on the original image. The cropping center of positive samples introduces a random offset near the center of the defect area to ensure sample diversity and simulate local visibility; negative samples randomly crop the background area that does not contain defects.

[0043] 2) Multi-resolution sampling: The original image is scaled according to a series of ratios (e.g., 1, 0.5, 0.25) to construct a sampling pyramid. At each level of the pyramid, the above positive and negative sample sampling is performed to automatically generate a training dataset with diverse scales and positive and negative balance.

[0044] 3) Labeling, Mapping, and Filtering: For samples that contain only partial defects after cropping, the intersection of the original defect polygon and the cropping box polygon is calculated, and the intersection coordinates are mapped to the sub-image. Simultaneously, an area threshold (e.g., 50 pixels) is set to filter out excessively small defect fragments, ensuring sample quality.

[0045] Step S103, Model Construction and Training: Construct an improved defect detection model based on the Mask R-CNN instance segmentation network; input the constructed dataset into the improved defect detection model for training to obtain a trained model.

[0046] In some embodiments, improvements to the model include: 1) Embedded coordinate attention module: A coordinate attention mechanism is embedded in the backbone network (such as ResNet50+FPN). This module generates orientation-aware feature maps by performing one-dimensional global pooling along the horizontal and vertical directions respectively, enhancing the model's ability to perceive the spatial location of defects.

[0047] 2) Introducing Focal Loss: Focal Loss is used to replace traditional cross-entropy loss in the classification task of region proposal networks. By adjusting the focus parameter γ and the weight coefficient α, the model pays more attention to samples that are difficult to classify (such as small defects and blurry defects) during training, thus alleviating the sample imbalance problem.

[0048] The improved model was trained using the dataset generated in step 102, and the optimal model weights were evaluated and selected using metrics such as mAP, precision, recall, and F1-score.

[0049] Step S104, Result Prediction and Output: The high-resolution image to be detected is segmented into sub-images through a sliding window and input into the trained model for prediction; the prediction results of the sub-images are mapped back to the original image coordinates, and the prediction results of adjacent masks belonging to the same defect category are fused based on connectivity analysis, and finally the defect category, location and mask information are output.

[0050] In some embodiments, predicting the high-resolution curtain wall image to be detected includes: 1) Sliding window prediction: A large image is divided into sub-images using a sliding window with a step size smaller than the window size. The sub-images are then input into a trained model for prediction, and the results are mapped back to the coordinates of the original image.

[0051] 2) Connectivity analysis and fusion: Connectivity analysis is performed on the prediction results mapped back to the original image to fuse spatially adjacent defect masks belonging to the same category into a single defect target, thus solving the problem of segmented detection of large defects.

[0052] 3) Multi-scale prediction pyramid: The image to be tested is constructed into a multi-scale pyramid with the same parameters as during training. Sliding window prediction and result fusion are performed independently at each level. Finally, the prediction results of all scales are merged to make full use of semantic and detail information at different scales. The final output is the defect category, bounding box, mask contour and confidence score.

[0053] The following is a detailed description and explanation of the embodiments of the present invention, in conjunction with the accompanying drawings and specific application examples.

[0054] This embodiment proposes a method for detecting defects in building curtain walls based on unmanned aerial vehicles (UAVs) and deep learning. The method includes the following steps: Step A, Image Acquisition: Acquire high-resolution RGB images of the building's curtain wall using a drone.

[0055] In some embodiments, step A specifically includes: using an industrial drone platform equipped with a gimbal camera to capture high-resolution RGB images of the curtain wall using an ultra-high-definition matrix mode. A DJI Matrice M30 drone is used as the flight platform, employing a gimbal camera with controllable left-right and pitch angles, and a controllable rotation range of translation. Looking up and down to It features a zoom camera with a 48-megapixel CMOS sensor and a lens with an equivalent focal length of 113-405mm. During image capture, DJI's built-in Ultra Matrix mode was used, with the drone positioned 10-15m vertically from the curtain wall, resulting in RGB images with a resolution of 4000×3000.

[0056] Step B, Data Preprocessing: Manually annotate the acquired high-resolution images once, and use a script to preprocess the annotated images, including positive and negative sample sampling and multi-resolution sampling, to automatically generate a curtain wall defect dataset of appropriate size and balanced positive and negative samples, and to realize data augmentation.

[0057] In some embodiments, step B specifically includes: using Labelme to perform pixel-level annotations on the original high-resolution image for defects such as broken glass, stone cracks, and sealant detachment in the building curtain wall. This method directly annotates the original image manually, and then uses a script to map the annotations to cropped smaller images, significantly improving annotation efficiency compared to annotating each smaller image individually. Then, a fixed-size cropping window (e.g., 800×800) is used to crop the original image, generating positive and negative samples to construct the dataset. Figure 2 As shown in the diagram. For positive samples, a random offset is added to the center of the cropping box near the center of the defect region, while ensuring that the cropped portion includes part of the defect. The defect annotations from the original image are mapped to the cropped smaller image. Multiple positive samples can be generated for the same defect, significantly increasing the dataset size given the limited original data. For negative samples, the cropping box randomly selects a background region that does not contain the defect, avoiding the problem of missing corresponding negative samples due to excessively large defects such as broken glass. The overall algorithm is as follows: Figure 3 As shown in the figure. This preprocessing also alleviates the problem of positive and negative sample imbalance in defect detection.

[0058] The aforementioned multi-resolution sampling takes into account the large size range of various defects in building curtain walls, and the fact that different defect types are suitable for prediction at different scales, such as... Figure 4 As shown. Therefore, a sampling pyramid is used to perform multi-resolution sampling of the original image, such as... Figure 5As shown, the original image is scaled a series of times, with scaling factors of 1, 0.5, and 0.25 used in the figure. Each scaled image is processed using the positive and negative sample sampling method described above to obtain positive and negative samples at different scales. The cropping frame can use the same size, such as... Figure 5 The cutting frames used are all 800×800, but manual design is also possible to better adapt to different defect types.

[0059] During the cropping process of the original image, positive and negative samples are processed to ensure the quality of the acquired samples. For positive samples, the cropping script may malfunction in the following situations: First, for defects with special shapes, such as C-shaped cracks, the cropping box center point is randomly offset from the bounding box center point, and the cropped sample may still not contain the defect. Second, the cropping box contains part of the defect. In this case, the Shapely library is used to convert the polygon annotations of the defect and the cropping box into polygons, and then the intersection function of the polygons is used to calculate the intersection between the two polygons. Then, the coordinates of the obtained intersection polygons are mapped onto the small image, thus completing the annotation mapping of part of the defect, such as... Figure 6 As shown. During this cropping process, some defect areas may be cropped to be too small, possibly even less than 20 pixels in the largest direction. Such positive samples are insufficient to reflect the characteristics of building defects and will mislead the subsequent learning of the network. Therefore, a filtering step is added to the script. Considering that crack defects themselves are relatively narrow, area is used as the filtering threshold. Only when the area of ​​a defect region in a positive sample is greater than 50 pixels will it be added to the generated dataset. For negative samples, at the highest level of the sampling pyramid, i.e., when the image is shrunk to its smallest size, if the defect area in the image is relatively large, such as... Figure 5 In the case of broken glass, the background area in the image may be smaller than the cropping box. To prevent changes in the aspect ratio or inconsistent sizes of samples in subsequent network iterations that could slow down training, black areas are used to fill the portion of the cropping box that exceeds the frame size. This same situation can occur when acquiring positive samples where the defect is located at the edge; in this case, black areas are also used for filling.

[0060] Step C, Construct a neural network model: Build an improved instance segmentation network based on Mask R-CNN, input the constructed dataset into the network for training, and output the network weights with the best evaluation metrics for prediction.

[0061] In some embodiments, step C specifically includes: constructing an instance segmentation network model based on Mask R-CNN, such as... Figure 7As shown, a coordinate attention module is added to the ResNet backbone network of Mask R-CNN, Focal Loss is introduced into RPN, and the constructed instance segmentation network model is trained.

[0062] The implementation of the coordinate attention module is as follows: Figure 8 As shown, assuming the input feature map has a shape of C×H×W, in the coordinate information embedding step, the module uses two one-dimensional global pooling operations (X Avg Pool and Y Avg Pool) to aggregate features in the horizontal and vertical directions, respectively, thereby generating a pair of orientation-aware feature maps (C×1×W and C×H×1) that preserve spatial location information. In the coordinate attention generation step, these feature maps are combined and passed to a shared 1×1 convolutional transformation function. After non-linear activation, they are split into two separate tensors. Two more 1×1 convolutional transformation functions are then used to generate horizontal and vertical attention maps, respectively. Finally, the product of the feature map and the attention map is output, enabling the network to more accurately locate objects of interest.

[0063] The expression of Focal Loss is shown below:

[0064] in It represents the probability of positive or negative samples predicted by the model, while γ is the focus parameter, which controls the degree of importance given to hard-to-classify samples. These are weighting coefficients used to balance the weights between categories. γ and γ were set to 0.25 and 0.2, respectively. Focal Loss allows the model to focus on hard-to-classify samples, avoiding overfitting to the background and easy-to-classify samples.

[0065] The training of the instance segmentation network includes setting neural network parameters and evaluating neural network training metrics.

[0066] The neural network parameters are set as follows: input image size is set to 800*800, batch size is set to 4, SGD is used as the optimizer, linear decaying learning rate scheduler is used, initial learning rate is set to 0.04, momentum parameter is 0.9, and batch size is set to 4.

[0067] The neural network training evaluation metrics are mAP, precision, recall, and F1-score; the calculation methods for these four metrics are as follows:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] Where TP indicates that the detected defect is actually a defect; TN indicates that the detected non-defect is actually a non-defect; FP indicates that the detected defect is actually a non-defect; and FN indicates that the detected non-defect is actually a non-defect. The true bounding box representing the defect, The predicted bounding box represents the defect; IOU represents the intersection-union ratio between the ground truth bounding box and the predicted bounding box. This indicates calculating the area of ​​the region within the parentheses; where n represents the number of categories. This represents the mAP value of defect i. This represents the AP value of defect i when IOU=0.5.

[0074] Step D, Result Prediction and Output: The high-resolution image to be predicted is segmented into smaller images, and the multi-resolution pyramid is combined with the sliding window. The prediction results of the smaller images are then post-processed to output the defect category, location information, and mask information, generating image annotations.

[0075] In some embodiments, step D specifically includes: when predicting high-resolution images captured by a drone, firstly, a 4000×3000 image is divided into smaller 800×800 images using a sliding window approach, with the step size of the sliding window slightly smaller than the window size of 750. Then, the smaller images are input into a trained neural network for prediction, and the prediction results are mapped back to the large 4000×3000 image, such as... Figure 9 As shown, this process allows the neural network to be trained and predicted using images of the same resolution, thus improving prediction accuracy. During the mapping of the prediction results back to the larger image, connectivity is determined. That is, if the defect masks in different smaller images are adjacent and belong to the same type of defect, then these predicted defect masks are merged into a single predicted defect. This approach can handle larger defects, such as broken glass, which are segmented into different smaller images during detection, affecting the detection results.

[0076] To address the issue of large size variations in building curtain wall defects and to maintain consistency with image sizes used during network training, a multi-scale pyramid is employed in the prediction phase, using the same scaling factor as the sampling pyramid. Figure 10As shown, each layer uses the sliding window cropping and post-processing steps described above. The prediction results from different layers are integrated using a designed algorithm; the figure uses the union method to obtain the final detection result. After the above process, the neural network will finally output the defect category, location information, and mask information, generating the image annotation result.

[0077] In summary, this application proposes a method for detecting defects in building curtain walls based on unmanned aerial vehicles (UAVs) and deep learning, including steps such as image acquisition, data preprocessing, neural network model construction, prediction post-processing, and output. By employing methods such as positive and negative sample acquisition, sampling pyramids, coordinate attention, Focal Loss, sliding windows, connectivity detection, and multi-size prediction, it provides new solutions to problems such as the difficulty of training and predicting high-resolution large-size images, limited sample numbers, large defect size spans, difficulty in detecting small-size defects, suppression of complex background interference, imbalanced positive and negative sample distribution, imbalanced sample classification, and repeated prediction of single defects. This improves the automation and accuracy of building curtain wall defect detection.

[0078] In summary, compared with the prior art, the method of this embodiment has at least the following advantages and beneficial effects: 1) High efficiency and low cost: Using drones to collect images is safe and efficient; adopting the "one-time annotation + automatic script processing" mode greatly reduces the manpower and time costs of pixel-level annotation.

[0079] 2) High accuracy and strong robustness: Through multi-resolution sampling and multi-scale prediction, the problem of detecting multi-scale defects in large images is effectively solved; the improved model (coordinate attention + Focal Loss) significantly improves the detection ability of small targets and difficult samples and the robustness in complex backgrounds.

[0080] 3) High engineering practicality: The entire process is highly automated, and the output results include pixel-level masks, which can intuitively guide maintenance work and have the potential for large-scale engineering applications.

[0081] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0082] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] Please see Figure 11 , Figure 11The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1102 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 using the methods described in the embodiments of this application. Input / output interface 1103 is used to implement information input and output; The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104); The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0084] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0085] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0086] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0088] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0089] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0090] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0093] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0094] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0096] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for intelligent detection of defects in building curtain walls, characterized in that, The method includes the following steps: Image acquisition: RGB images of the building's curtain wall are acquired using a high-resolution camera mounted on a drone; Data preprocessing: The acquired high-resolution image is annotated at the pixel level in one go; then, the data preprocessing is automatically performed by a script. The preprocessing includes: sliding cropping of the original annotated image based on a fixed-size cropping window to generate positive samples containing defective regions and negative samples without defective regions, and constructing a dataset with balanced positive and negative samples; at the same time, the original image is scaled at multiple resolutions to construct a sampling pyramid, and the positive and negative sample sampling is performed at each layer of the pyramid to achieve data augmentation and multi-scale dataset construction. Model building and training: An improved defect detection model is built based on the Mask R-CNN instance segmentation network; the constructed dataset is input into the improved defect detection model for training to obtain a trained model; Result prediction and output: The high-resolution image to be detected is segmented into sub-images through a sliding window and input into the trained model for prediction; the prediction results of the sub-images are mapped back to the coordinates of the original image, and the prediction results of adjacent masks belonging to the same defect category are fused based on connectivity analysis, and finally the defect category, location and mask information are output. The process of generating positive samples includes: using the center of the defect area as a reference, introducing a random offset to the center point of the clipping window to ensure that the clipping window contains at least part of the defect area; The process of generating negative samples includes: randomly selecting a background region in the original image that does not contain any defective areas and cropping it; For positive samples that contain only part of the defect area after cropping, polygon intersection operation is used to map the original annotations to the cropped sub-image; and an area threshold is set to filter out positive samples with too small a defect area after mapping. At the top layer of the sampling pyramid, i.e. the layer with the smallest image size, when the background area of ​​the image is smaller than the fixed-size cropping window, a padding method is used to ensure that the output sample size is consistent. The improvements to the improved defect detection model include: A coordinate attention module is embedded in the backbone network of Mask R-CNN; In the region proposal network of Mask R-CNN, Focal Loss is used as the classification loss function; During the prediction phase, a multi-resolution prediction pyramid with the same scaling factor as that used in the training phase is constructed. The image to be detected is scaled to each layer of the prediction pyramid, and the sliding window prediction and result mapping are performed independently at each layer; Finally, the prediction results of all layers of the pyramid are merged to form the final defect detection result.

2. The method according to claim 1, characterized in that, The execution process of the coordinate attention module includes: Coordinate information embedding: Perform one-dimensional global pooling in the width and height directions on the input feature map to generate a pair of orientation-aware feature maps; Coordinate attention generation: After concatenating and transforming the orientation-aware feature map, it is divided into two independent tensors, and attention maps in the width and height directions are generated by convolution transformation respectively. The generated attention map is multiplied with the original input feature map to enhance the model's ability to perceive spatial location.

3. The method according to claim 1, characterized in that, The functional expression for Focal Loss is: in, It is the model's predicted probability for positive and negative samples. It is a coefficient used to balance class weights. γ It is a focusing parameter used to adjust the weights of easy and difficult samples.

4. The method according to claim 1, characterized in that, The step size of the sliding window is smaller than the window size to ensure that defects in the window edge area can be effectively detected. The connectivity analysis specifically involves determining whether the predicted defect masks in different subgraphs are spatially adjacent and belong to the same defect category. If so, the multiple prediction results are merged into a single defect target.

5. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images