Multi-mode layered cross fusion unmanned aerial vehicle underground environment crack detection method

Abstract

The invention discloses a multi-mode layered cross fusion unmanned aerial vehicle underground environment crack detection method. The method comprises the following steps: extracting depth features based on a visible light image, an infrared image and a laser radar point cloud; fusing the extracted depth features to obtain fused features; and segmenting the fusion features through a segmentation network to obtain segmented crack masks, and outputting a crack detection result through a classification network. According to the multi-mode layered cross fusion unmanned aerial vehicle underground environment crack detection method disclosed by the invention, the expression ability of crack edges and textures is enhanced, and the crack detection accuracy is improved.

Description

A Multimodal Layered Cross-fusion Method for Unmanned Aerial Vehicle (UAV) Crack Detection in Underground Environments Technical Field

[0001] This invention relates to a multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs), belonging to the field of UAV control technology. Background Technology

[0002] In underground environments such as mines and tunnels, using drones for dynamic inspection of hazardous sources is an efficient and safe method. Among these, crack detection is a crucial task for identifying potential geological hazards and preventing them in advance. However, the underground environment is characterized by insufficient lighting, high dust concentration, and confined space, resulting in generally poor image quality acquired by drones, which poses a significant challenge to detection algorithms.

[0003] First, when drones inspect underground environments, they often rely solely on their own light sources or limited fixed lighting. The lighting conditions vary drastically with position and angle, leading to underexposure or strong reflections in visible light images, significantly reducing the clarity of crack textures and edges. Second, high concentrations of dust are frequently present during mining operations, further blurring images and reducing contrast, causing crack features to be partially obscured or distorted. Moreover, cracks themselves are characterized by large scale variations, irregular shapes, and complex spatial distribution. As the drone's flight altitude and distance change, the scale of cracks in the image exhibits strong inconsistencies, increasing the difficulty of detection.

[0004] However, existing detection methods are significantly inadequate under the aforementioned complex environmental conditions. For example, single-modal image detection methods, such as segmentation methods based on DeepLabv3+ and the Segformer architecture, achieve good results on standard or well-lit datasets. However, in underground scenes, image quality degradation makes the models lack robustness to fine-grained crack features (edges, textures), leading to high false negative and high false positive rates. On the other hand, while existing multimodal crack detection methods fuse crack geometric information (such as width, direction, and location), most are still based on simple extensions of single-modal features and lack effective mechanisms to handle complex interference factors such as drastic changes in lighting, dust occlusion, and inconsistent scale. Therefore, their performance improvement in real-world underground scenes is limited. Furthermore, multimodal fusion methods are currently mainly applied to autonomous driving and ground scene perception, and research on underground inspection environments remains lacking, failing to meet the accuracy and reliability requirements of dynamic inspection of hazardous sources.

[0005] Therefore, it is necessary to conduct more in-depth research on existing crack detection methods, especially those for complex underground environments, in order to solve the above problems. Summary of the Invention

[0006] To overcome the above problems, in-depth research was conducted, and a multimodal hierarchical cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) was proposed, including the following steps: S1, extracting depth features based on visible light images, infrared images, and lidar point clouds respectively; S2, fusing the extracted depth features to obtain fused features; S3, segmenting the fused features through a segmentation network to obtain segmented crack masks, and outputting crack detection results through a classification network.

[0007] In a preferred embodiment, in S1, depth features are extracted using a multi-scale residual dense network. The multi-scale residual dense network includes residual dense connection blocks and cascaded multi-scale dilated convolutional blocks. The residual dense connection blocks are formed by densely connecting multiple convolutional layers with residuals and then... Convolutional layers are formed, wherein the cascaded multi-scale dilated convolutional blocks are multiple layers of dilated convolutions connected in parallel and then processed. Convolutional layers are formed.

[0008] In a preferred embodiment, the output of the multi-scale residual dense network is represented as follows: ,in, This represents the output of a multi-scale residual dense network. This represents the output of the residual dense connection block. This represents the output of a cascaded multi-scale dilated convolution block.

[0009] In a preferred embodiment, the visible light image and infrared image are further subjected to a convolutional layer to unify the feature dimensions before depth feature extraction.

[0010] In a preferred embodiment, a bird's-eye view is generated based on the lidar point cloud before depth feature extraction.

[0011] In a preferred embodiment, the hierarchical cross-attention mechanism is based on the extracted deep features. It uses two attention mechanisms, multi-head attention and cross-attention, to globally exchange intermediate features, concatenate the features in the attention mechanism, and then generate new modal features based on residual connections.

[0012] In a preferred embodiment, in S3, the segmentation network is preferably a lightweight Segformer segmentation network.

[0013] The beneficial effects of this invention include: 1) It uses dense residual connections and multi-scale dilated convolutional layers to extract depth features, simultaneously extracting local texture and global structural information of cracks under different receptive fields. Compared with traditional crack segmentation algorithms, this method has richer crack feature representation and robustness, improving the accuracy of crack detection; 2) It introduces a multimodal feature fusion mechanism, jointly modeling visible light images with infrared and lidar information, overcoming the defect of single-modal imaging quality degradation in harsh environments such as low light and high dust; it achieves complementary fusion of multiple modal information through a hierarchical cross-attention mechanism, enhancing the ability to express crack edges and textures; 3) It adopts the lightweight segmentation framework Segformer, which can run on UAV airborne platforms with limited video memory and computing resources, and maintains inference speed under excellent detection accuracy conditions, thus meeting the requirements of high-precision detection and real-time performance in actual inspection scenarios. Attached Figure Description

[0014] Figure 1 shows a flowchart of a UAV underground environment crack detection method based on a preferred embodiment of the present invention, which employs multimodal hierarchical cross-fusion; Figure 2 shows a structural diagram of a UAV underground environment crack detection method based on a preferred embodiment of the present invention; Figure 3 shows a schematic diagram of a multi-scale residual dense network structure of a UAV underground environment crack detection method based on a preferred embodiment of the present invention; Figure 4 shows a schematic diagram of a hierarchical cross-attention mechanism of a UAV underground environment crack detection method based on a preferred embodiment of the present invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.

[0016] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0017] The present invention provides a multimodal hierarchical cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs), as shown in Figures 1 and 2, comprising the following steps: S1, extracting depth features based on visible light images, infrared images, and lidar point clouds respectively; S2, fusing the extracted depth features to obtain fused features; S3, segmenting the fused features using a segmentation network to obtain segmented crack masks, and outputting crack detection results using a classification network.

[0018] In S1, depth features are extracted using a multi-scale residual dense network. This multi-scale residual dense network includes residual dense connective blocks and cascaded multi-scale dilated convolutional blocks. The residual dense connective blocks are formed by densely connecting multiple convolutional layers with residuals, followed by... The convolutional layers are formed, as shown in Figure 3. The dense residuals between the convolutional layers are represented as follows:

[0019] in, This represents the output of the i-th convolutional layer. This represents the activation function. These are the weights of the i-th convolutional layer. This is the input to the first convolutional layer. This is the connection between the input of the first convolutional layer and the outputs of the first to (i-1)th convolutional layers.

[0020] According to the present invention, a method is set after the residual dense connection block. Convolutional layers adaptively control the output information, represented as:

[0021] in, This represents the output of the convolutional layer. express Convolutional layer.

[0022] The cascaded multi-scale dilated convolutional block consists of multiple parallel dilated convolutions followed by... The convolutional layer is formed, as shown in Figure 3. The cascaded multi-scale dilated convolutional block is represented as follows:

[0023] in, This represents the output of a cascaded multi-scale dilated convolution block. express Operations of convolutional layers, This represents the output of the j-th dilated convolution layer connected in parallel.

[0024] The output of the multi-scale residual dense network is represented as follows:

[0025] in, This represents the output of a multi-scale residual dense network.

[0026] Traditional feature extractors, such as the U-Net encoder, typically downsample the image by a factor of 16, and then perform a series of upsampling steps to restore the original image. However, when dealing with the edges and breakpoints of small cracks, excessive downsampling can easily lead to the loss of high-frequency detail signals. Furthermore, continuous convolution can blur small cracks and fail to preserve shallow features within deep features, resulting in a lack of multi-scale detection capabilities in the model.

[0027] In this invention, a residual dense network is used to ensure that shallow edge texture features can reach all subsequent layers, and the global contour is preserved through the residual structure. A cascaded dilated convolutional layer is used to simultaneously acquire local details (fine cracks) and global context (wide cracks, breakpoint connections) in the same layer through different receptive fields, maintaining the integrity of features at different scales and realizing multi-scale detection capability, thereby solving the problem of edge feature and fine feature loss.

[0028] Furthermore, the multi-scale residual dense network designed in this invention can make full use of the hierarchical features of each convolutional layer to extract rich local features, and obtain multi-scale information through cascaded multi-layer dilated convolutions to achieve simultaneous response to cracks of multi-scale width.

[0029] Furthermore, the visible light image and the infrared image are images from the same time and location.

[0030] Preferably, before depth feature extraction, the visible light image and infrared image are subjected to a convolutional layer to unify the feature dimensions, and the nonlinearity of the features is enhanced by the convolutional layer processing.

[0031] Preferably, before performing depth feature extraction, the lidar point cloud is first used to generate a bird's-eye view (BEV) to transform the sparse point cloud into regular two-dimensional features, which facilitates alignment and fusion with features from visible light images and infrared images.

[0032] The bird's-eye view is a two-dimensional grid formed by encoding height and density.

[0033] Preferably, the projected point cloud is discretized into a two-dimensional grid with a resolution of 0.1 meters.

[0034] Preferably, the point cloud is divided into M equal parts along the height direction, a height map is calculated for each part, resulting in M ​​height maps, and the height information is encoded using the maximum value of each height map.

[0035] The density is expressed as:

[0036] in, The value represents density, and N represents the number of point clouds within a cell.

[0037] Preferably, the point cloud is divided into 5 equal parts along the height direction, and the final bird's-eye view contains 6 channels. The first 5 channels are the maximum height values ​​corresponding to the 1st to 4th equal parts, the 5th channel is the maximum height value in the whole point cloud, and the 6th channel represents the point cloud density information.

[0038] In S2, the fusion is performed using a hierarchical cross-attention mechanism.

[0039] In the hierarchical cross-attention mechanism, based on the extracted deep features, the intermediate features are globally exchanged using two attention mechanisms: multi-head attention and cross-attention. The features in the attention mechanism are concatenated together, and then new modal features are generated based on residual connections, as shown in Figure 4.

[0040] The hierarchical cross-attention mechanism is represented as follows:

[0041]

[0042] ,

[0043] in, This represents the output of the hierarchical cross-attention mechanism. , , , As an intermediate feature, Represents depth features extracted from visible light images. The inherent characteristics, Represents depth features extracted from infrared images. The inherent characteristics, This represents the depth features extracted from the point cloud by lidar. The key, This represents the depth features extracted from the point cloud by lidar. The value, Represents depth features extracted from infrared images The key, Represents depth features extracted from infrared images The value, Indicate intermediate features The query, Indicate intermediate features The key, Indicate intermediate features The value of .

[0044] According to the present invention, by using a hierarchical cross-attention mechanism, the advantages of three modalities are fused based on the principle of complementarity, which makes up for the defects of visible light images in environments with severe image degradation and expands the application scenarios.

[0045] In S3, the segmentation network preferably employs a lightweight Segformer segmentation network, which combines a hierarchical Transformer encoder with a lightweight fully connected decoder. This design allows for the extraction of multi-scale contextual information while maintaining low computational complexity. Using the Segformer segmentation network, fine segmentation of crack regions can be achieved, outputting pixel-level crack masks. This lightweight segmentation framework allows operation on UAV platforms with limited GPU memory and computational resources, maintaining inference speed while achieving excellent detection accuracy. It can balance high-precision detection with real-time requirements in practical inspection scenarios.

[0046] In this invention, the specific structure of the classification network is not limited, and those skilled in the art can use any known classification network.

[0047] Example 1: Crack detection experiment was conducted. Crack detection includes the following steps: S1: Extract depth features based on visible light image, infrared image and lidar point cloud respectively; S2: Fuse the extracted depth features to obtain fused features; S3: Segment the fused features through a segmentation network to obtain segmented crack masks, and output crack detection results through a classification network.

[0048] In S1, depth features are extracted using a multi-scale residual dense network. This multi-scale residual dense network includes residual dense connective blocks and cascaded multi-scale dilated convolutional blocks. The residual dense connective blocks are formed by densely connecting multiple convolutional layers with residuals, followed by... Convolutional layers are formed, wherein the cascaded multi-scale dilated convolutional blocks are multiple layers of dilated convolutions connected in parallel and then processed. With the formation of convolutional layers, the output of the multi-scale residual dense network is represented as:

[0049] In S2, the fusion is performed using a hierarchical cross-attention mechanism, which is expressed as follows:

[0050]

[0051] ,

[0052] In S3, the segmentation network uses a lightweight Segformer segmentation network.

[0053] Experiments were conducted using a dataset that collected visible light images, infrared images, and lidar point cloud data under various lighting conditions, dust blurring, and different underground environments.

[0054] Comparative Example 1 uses the same dataset as Example 1 for experiments, except that it uses the DeepLabv3+ method and the Segformer baseline method respectively.

[0055] The results of Comparative Example 1 and Comparative Example 1 were compared, with the mean intersection-to-union ratio (mIOU) and mean accuracy (mACC) used as evaluation criteria, as shown in Table 1.

[0056] Method Average Precision Average Cross-Union Comparison Example 1 - DeepLabv3+ 48.5% 42.8% Comparative Example 1 - Segformer 6 1.3% 56.3% Example 1 8 1.8% 77.8% surface

[0057] As can be seen from Table 1, the method in Example 1 has significantly improved the mean intersection-to-union ratio (mIOU) and mean accuracy (mACC), and the crack detection capability has been improved by more than 20%, enabling UAVs to inspect cracks in underground environments.

[0058] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front," and "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship in the working state of this invention, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0059] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0060] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.

Claims

1. A multimodal, hierarchical, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: S1. Extract depth features based on visible light images, infrared images, and lidar point clouds, respectively; S2. The extracted deep features are fused to obtain fused features; S3. The fused features are segmented by a segmentation network to obtain segmented crack masks, and crack detection results are output by a classification network.

2. The multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, In S1, depth features are extracted using a multi-scale residual dense network. This multi-scale residual dense network includes residual dense connective blocks and cascaded multi-scale dilated convolutional blocks. The residual dense connective blocks are formed by densely connecting multiple convolutional layers with residuals, followed by... Convolutional layers are formed, wherein the cascaded multi-scale dilated convolutional blocks are multiple layers of dilated convolutions connected in parallel and then processed. Convolutional layers are formed.

3. The multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The output of the multi-scale residual dense network is represented as follows: ,in, This represents the output of a multi-scale residual dense network. This represents the output of the residual densely connected block. This represents the output of a cascaded multi-scale dilated convolution block.

4. The multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Before performing depth feature extraction, the visible light and infrared images are also subjected to a convolutional layer to unify the feature dimensions.

5. The multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Before performing depth feature extraction, the lidar point cloud is first used to generate a bird's-eye view.

6. The multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, In the hierarchical cross-attention mechanism, based on the extracted deep features, the intermediate features are globally exchanged using two attention mechanisms: multi-head attention and cross-attention. The features in the attention mechanisms are then concatenated, and new modal features are generated based on residual connections.

7. The multimodal, layered, cross-fusion method for detecting cracks in underground environments using unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, In S3, the segmentation network preferably adopts the lightweight Segformer segmentation network.

Images