Multi-sensing information fusion old building safety detection method based on unmanned aerial vehicle

Abstract

The invention discloses a multi-sensing information fusion old building safety detection method based on an unmanned aerial vehicle, and the method comprises the steps: obtaining three-dimensional point cloud data of a building and a surrounding environment, and constructing an old building three-dimensional model through the three-dimensional point cloud data; performing defect detection on the old building outer wall to obtain defect positions and shape types of the old building outer wall; detecting the old building to obtain a temperature abnormal area of the old building; and carrying out safety detection on the old building by fusing the old building three-dimensional model, the old building outer wall defect position, the shape type and the old building temperature anomaly region. The method greatly improves the detection efficiency, precision and safety, and has practical significance and popularization value.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) remote sensing nondestructive testing, and in particular to an old building safety testing method based on unmanned aerial vehicle (UAV) multi-sensor information fusion. Background Art

[0002] Old buildings that have been in use for a long time may pose serious safety hazards and require regular inspection and maintenance. At present, the existing building deformation monitoring and addition, modification and expansion identification on the market are mainly achieved by manual inspection or deployment of sensors. These methods have high costs, large errors, cannot be adopted on a large scale, and cannot detect abnormal conditions of old houses in time. In contrast, multi-sensor information fusion technology based on drones is an efficient, non-contact and widely used modern technical means. It can collect and analyze data of old buildings comprehensively and accurately through the laser radar (LiDAR), high-resolution camera and thermal imager carried by drones, so as to evaluate their structural safety, hidden danger areas and potential risks. The use of drones and multi-sensor information fusion technology to conduct safety inspections on old buildings can greatly improve the efficiency, accuracy and safety of inspections.

[0003] The multi-sensor information fusion safety inspection method for old buildings based on drones mainly includes the safety inspection method for old buildings using sensors, such as patent CN118811139A, which uses drones equipped with target detection cameras and infrared thermal imagers to inspect the facades of high-rise buildings; patent CN119007039A, which uses drone aerial photos and YOLOv8-based structural similarity detection to compare the similarity of changed areas between new and old buildings; patent CN117419813A, which uses drones to collect thermal imaging information of building exterior walls and use it for building exterior wall quality inspection.

[0004] The above-mentioned specific patent comparative documents are:

[0005] 1) "A high-rise building facade detection device based on drone", patent number CN118811139A. This patent relates to a high-rise building facade detection device based on drone, which relates to the field of high-rise building detection technology, including a drone body, a target detection camera and an infrared thermal imager installed on the top of the drone body, and arms arranged in a ring on the four sides of the drone body, and propellers are installed on the arms. The facade support plate can be quickly disassembled and assembled on the arms. The facade support plate is not only easy to install and replace, but also helps to support and protect the propeller. The facade support plate is rotatably connected to the bottom of the facade support plate. The movable adsorption plate forms an unfolded cladding state and a retracted support state under the telescopic control of the movable telescopic rod; when the movable adsorption plate is in the unfolded cladding state, on the one hand, the air pump is used to blow air from the negative pressure suction cup to blow away the dust on the surface of the building facade, and on the other hand, the air pump is used to suck air from the negative pressure suction cup, so that the movable adsorption plate can be stably adsorbed on the building facade, thereby ensuring that the entire drone is stabilized on the outside of the high-rise building facade. The present invention uses information fusion of three sensors including lidar, high-resolution camera and thermal imager to carry out safety inspection of old buildings. By comparing the two sensors, camera and infrared thermal imager, used in this method, more comprehensive information can be obtained.

[0006] 2) "A method for detecting illegal buildings based on drone image comparison", patent number CN118811139A. This patent relates to a method for detecting illegal buildings based on drone image comparison, and the detection method comprises the following steps: step S1, collecting images through drones, collecting aerial photos of various streets; step S2, preprocessing the collected images; step S3, performing background difference on the preprocessed images, extracting three-channel color components from the images, performing difference operations and threshold processing on the new and old phases of the same color component, and detecting the image change area; step S4, applying the deep learning classification algorithm to the task of image classification of the changed area, and excluding image changes caused by non-construction reasons; step S5, using structural similarity detection to compare the similarity of the changed areas of the new and old phases. The present invention uses laser radar to reconstruct the three-dimensional structure of old buildings, which has higher accuracy than the image-based structural similarity detection used by this method.

[0007] 3) "A method for detecting the quality of building exterior walls based on big data", patent number CN117419813A. This patent relates to the field of building detection technology, and in particular to a method for detecting the quality of building exterior walls based on big data, including: building an engineering management platform and a job site collection platform, setting up the engineering management platform for remotely sending BIM data and remotely receiving thermal imaging models; the job site collection platform is provided with an industrial computer and at least one drone for data interaction with the industrial computer, the industrial computer and the engineering management platform perform remote information interaction, and the drone and the industrial computer perform close-range data interaction to ensure the accuracy of thermal imaging information collection of building exterior walls. The present invention uses information fusion of three sensors including laser radar, high-resolution camera and thermal imager to conduct safety inspections on old buildings. Compared with the thermal imager sensor used in this method, more comprehensive information can be obtained. Summary of the invention

[0008] In order to solve the above technical problems, the purpose of the present invention is to provide a multi-sensor information fusion old building safety detection method based on unmanned aerial vehicle.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] A method for safety detection of old buildings based on multi-sensor information fusion of unmanned aerial vehicles, the method is based on the fusion of unmanned aerial vehicles and multi-sensor information, and the method comprises the following steps:

[0011] Step A obtains 3D point cloud data of the building and its surrounding environment, and constructs a 3D model of the old building through the 3D point cloud data;

[0012] Step B performs defect detection on the exterior walls of old buildings to obtain the locations, shapes and types of defects on the exterior walls of old buildings;

[0013] Step C detects old buildings and obtains abnormal temperature areas of old buildings;

[0014] Step D integrates the three-dimensional model of the old building, the location, shape and type of defects on the exterior walls of the old building, and the abnormal temperature areas of the old building to conduct safety inspections on the old building.

[0015] Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

[0016] The three-dimensional point cloud data of the building and the surrounding environment obtained by LiDAR is used to construct a three-dimensional model of the old building. The model can be used to analyze safety issues such as structural deformation and foundation settlement of old buildings, and as the basis for multi-sensor information fusion. Defect detection of the exterior walls of old buildings is performed through a neural network model based on an encoder-decoder and a shifted window attention mechanism. A thermal imager carried by a drone is used to detect abnormal temperature areas of old buildings and identify safety issues of old buildings such as leakage and poor insulation. Finally, the information collected by the three sensors is fused, and the location, shape and type of defects on the exterior walls of old buildings obtained by a high-resolution camera and the temperature abnormality area obtained by the thermal imager are marked in the three-dimensional model constructed based on the data obtained by LiDAR. The use of the multi-sensor information fusion old building safety detection method based on drones of the present invention can greatly improve detection efficiency, accuracy and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the safety inspection method for old buildings based on multi-sensor information fusion based on drones;

[0018] Figure 2 It is a schematic diagram of the laser radar scanning method;

[0019] Figure 3 It is a schematic diagram of the network structure of high-resolution camera defect detection;

[0020] Figure 4 This is a schematic diagram of thermal imager imaging;

[0021] Figure 5 It is a multi-sensor information fusion flow chart. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with embodiments and drawings.

[0023] like Figure 1 As shown in the figure, a multi-sensor information fusion old building safety detection method based on drones includes:

[0024] Step 10: Obtain 3D point cloud data of the building and its surrounding environment, and construct a 3D model of the old building through the 3D point cloud data;

[0025] Step 20: perform defect detection on the exterior walls of old buildings to obtain the locations, shapes and types of defects on the exterior walls of old buildings;

[0026] Step 30 detects old buildings and obtains temperature abnormality areas of old buildings;

[0027] Step 40 integrates the three-dimensional model of the old building, the location, shape and type of defects on the exterior walls of the old building, and the abnormal temperature areas of the old building to conduct safety inspections on the old building.

[0028] The above step 10 specifically includes: using the 3D point cloud data of the building and the surrounding environment obtained by LiDAR to construct a 3D model of the old building, the scanning method is as follows: Figure 2 This model can be used to analyze safety issues such as deformation of old building structures and foundation settlement, and can serve as a basis for multi-sensor information fusion.

[0029] The above step 20 specifically includes: performing defect detection on the exterior walls of old buildings using a neural network model.

[0030] The neural network model is an encoder-decoder network with residual connections. The encoder extracts features from the images obtained by the high-resolution camera carried by the drone, and the decoder uses the features extracted by the encoder to detect defects in the exterior walls of old buildings. Figure 3 As shown in Figure 1. The image initially passes through a 3×3 convolutional layer to generate a low-level feature map. The feature map is then fed into a series of four encoder layers, each consisting of a Swin Transformer Layer (STL) block and a downsampling convolutional layer. The downsampling operation halves the spatial resolution of the feature map while doubling the number of channels, so that the feature maps at the output of the encoder layer have 64, 128, 256, and 512 channels, respectively. An intermediate layer is added to the end of the encoder sequence, which also contains an STL block. This layered structure enables the network to capture more comprehensive image features. After the intermediate layer, the feature map passes through four decoder layers, each of which includes an upsampling convolutional layer and an STL block. Before the feature maps are fed into the STL blocks in the decoder, they are connected to their corresponding encoder features through skip connections. The skip connection mechanism promotes the fusion of low-level details with high-level semantic information, enhancing the network's ability to reconstruct clear images. After the decoder layer, the feature map is further processed by a final convolutional layer to obtain the detection results of defects on the exterior walls of old buildings. The combination of STL blocks and the encoder-decoder architecture enables the network to effectively capture local and global contextual information, thereby improving the detection performance of defects on the exterior walls of old buildings.

[0031] The Swin Transformer Layer (STL) block structure is based on the multi-head self-attention mechanism of the original Transformer block, but the main difference lies in the local attention and shift window mechanism, such as Figure 3 Assume that the size of the input feature map is H×W×C, the STL block first divides the feature map into M×M non-overlapping local windows and reshapes it into a window of size (HW / M 2)×M×C feature map. The number of local windows is (HW / M 2 ). Then, self-attention operation, or local attention, is performed on each window. For a local window feature The calculation of the matrices Q, K and V corresponding to the query, key and value is shown in formula (1):

[0032] Q = XP Q ,K=XP K ,V=XP V (1)

[0033] Among them, P Q , P K and P V is the projection matrix shared between different windows. In general, Therefore, the attention matrix of the local window is calculated through the self-attention mechanism, as shown in the following formula (2):

[0034] Attention(Q,K,V)=SoftMax(QK T / d+B)V (2)

[0035] Where B represents the learnable relative position encoding. In this model, the attention function is replicated 8 times in parallel and the results are concatenated as a multi-head self-attention (MSA) operation.

[0036] After the MSA operation, a multi-layer perceptron (MLP) is used for further feature transformation. The MLP includes two fully connected layers and a GELU nonlinear transformation layer. A layer normalization (LN) layer is added before both MSA and MLP, and residual connections are added to both modules. The whole process is shown in formula (3):

[0037]

[0038] In order to solve the problem that the network’s ability to capture and utilize information from different spatial regions of the image is limited due to the lack of connection between local windows when the partitions of different layers are fixed and unchanged, the STL block uses a cross-window connection mechanism and alternates between regular windows and a shifted window mechanism. The shifted window mechanism shifts the features by (M / 2,M / 2) pixels before performing old building defect detection.

[0039] The above step 30 specifically includes: using the thermal imager carried by the drone to detect abnormal temperature areas in old buildings, and identifying safety issues of old buildings such as leakage and poor insulation. The imaging diagram is shown in the figure Figure 4 shown.

[0040] The above step 40 specifically includes: fusing the information collected by the three sensors. The multi-sensor information fusion flow chart is as follows: Figure 5 In the 3D model constructed based on the LiDAR data, the locations and shapes of defects on the exterior walls of old buildings obtained by high-resolution cameras and the temperature anomaly areas obtained by thermal imaging cameras are marked.

[0041] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A multi-sensor information fusion old building safety detection method based on drones, characterized in that: The method is implemented based on the fusion of drone and multi-sensor information, and the method includes the following steps: Step A obtains 3D point cloud data of the building and its surrounding environment, and constructs a 3D model of the old building through the 3D point cloud data; Step B performs defect detection on the exterior walls of old buildings to obtain the locations, shapes and types of defects on the exterior walls of old buildings; Step C detects old buildings and obtains abnormal temperature areas of old buildings; Step D integrates the three-dimensional model of the old building, the location, shape and type of defects on the exterior walls of the old building, and the abnormal temperature areas of the old building to conduct safety inspections on the old building.

2. The multi-sensor information fusion old building safety detection method based on drone according to claim 1 is characterized in that: In step A, the LiDAR carried by the drone is used to obtain three-dimensional point cloud data of the building and its surrounding environment; the three-dimensional model of the old building constructed using the three-dimensional point cloud data is used to analyze the structural deformation and foundation settlement problems of the old building, and serves as the basis for multi-sensor information fusion.

3. According to claim 1, the multi-sensor information fusion old building safety detection method based on drone is characterized in that: In step B, a high-resolution camera carried by a drone is used to detect defects on the exterior walls of old buildings through a neural network model; The neural network model is an encoder-decoder network with residual connections. The encoder extracts features from images obtained by a high-resolution camera carried by a drone, and the decoder uses the features extracted by the encoder to detect defects in the exterior walls of old buildings.

4. According to claim 3, the multi-sensor information fusion old building safety detection method based on drone is characterized in that: The network structure of the neural network model is as follows: the image initially generates a low-level feature map through a 3×3 convolution layer, which is input into a series of four encoder layers, each of which consists of an STL block and a downsampling convolution layer; the downsampling operation halves the spatial resolution of the feature map and doubles the number of channels, so that the feature maps at the output of the encoder layer have 64, 128, 256, and 512 channels respectively; An intermediate layer is added to the end of the encoder sequence, which also contains an STL block; after the intermediate layer, the feature map includes four decoder layers, each of which includes an upsampling convolutional layer and an STL block; after the decoder layer, the feature map is further processed by a final convolutional layer to obtain the detection results of defects on the exterior walls of old buildings.

5. According to claim 4, the multi-sensor information fusion old building safety detection method based on drone is characterized in that: The STL block structure is based on the multi-head self-attention mechanism of the Transformer block, the difference lies in the local attention and shift window mechanism. Assuming that the size of the input feature map is H×W×C, the STL block first divides the feature map into M×M non-overlapping local windows and reshapes it into a window of size (HW / M 2 )×M×C feature map; the number of local windows is (HW / M 2 ), then, perform self-attention operation on each window, or local attention; for a local window feature The calculation of matrices Q, K and V corresponding to the query Query, key Key and value Value is shown in formula (1): Q=XP Q ,K=XP K ,V=XP V (1) Among them, P Q , P K and P V is the projection matrix shared between different windows; where Therefore, the attention matrix of the local window is calculated through the self-attention mechanism, as shown in the following formula (2): Attention(Q,K,V)=SoftMax(QK T / d+B)V (2) Among them, B represents the learnable relative position encoding; after the attention function is replicated 8 times in parallel, the results are connected as the multi-head self-attention MSA operation.

6. The multi-sensor information fusion old building safety detection method based on drone according to claim 5 is characterized in that: After the MSA operation, a multi-layer perceptron MLP is used for further feature transformation; the MLP includes two fully connected layers and a GELU nonlinear transformation layer; and a normalized LN layer is added before both MSA and MLP, and a residual connection is added between MSA and MLP; The whole process is shown in formula (3): The STL block uses a cross-window connection mechanism and alternately uses a regular window and a shifted window mechanism; the shifted window mechanism shifts the feature by (M / 2, M / 2) pixels before performing old building defect detection.

7. The multi-sensor information fusion old building safety detection method based on drone according to claim 1 is characterized in that: In step C, a thermal imager carried by a drone is used to detect abnormal temperature areas in old buildings to identify leakage and poor insulation problems.

8. The multi-sensor information fusion old building safety detection method based on drone according to claim 1 is characterized in that: In the step D, the three-dimensional model of the old building, the location, shape and type of defects on the exterior walls of the old building, and the temperature abnormality area of ​​the old building are integrated. The three-dimensional model constructed based on the data obtained by LiDAR is marked with the location, shape and type of defects on the exterior walls of the old building obtained by a high-resolution camera and the temperature abnormality area obtained by a thermal imager.

Images