A deep learning-based cross-modal pipeline defect detection method and system

Abstract

The application provides a kind of based on deep learning's cross-modal pipeline defect detection method and system, comprising: obtaining the RGB image and depth image of pipeline to be detected, and using double-branch depth detection network to respectively carry out feature extraction to RGB image and depth image, obtain RGB feature and depth feature;In the high-level semantic layer of double-branch depth detection network, using feature fusion adapter, the RGB feature with the depth feature is carried out multi-modal feature alignment and adaptive weighted fusion, and the fusion feature is obtained;Based on the fusion feature, through detection head, realize the defect detection of pipeline.The application realizes the real-time, accurate identification and positioning of defect in resource-constrained industrial scene.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a method and system for cross-modal pipeline defect detection based on deep learning. Background Technology

[0002] As critical infrastructure for transporting oil, gas, water, and chemical media, pipelines' structural integrity directly impacts production safety, environmental protection, and economic benefits. However, due to aging, neglect, and lack of maintenance, in-service pipelines commonly suffer from leaks, corrosion, cracks, deformation, and weld defects, leading to frequent catastrophic accidents such as leaks, explosions, and road collapses, potentially causing significant economic losses and social impacts. Traditional pipeline defect detection relies heavily on manual inspections and experience-based judgment, lacking automated and intelligent dynamic sensing and early warning capabilities, making it difficult to achieve real-time monitoring and risk assessment of operational status. Furthermore, existing inspection resources are overly concentrated on main pipeline networks, neglecting differentiated and refined inspections of high-risk areas such as intersections, aging pipelines, buried culverts, and stormwater drainage systems, resulting in long-term latent hazards and frequent accidents. Therefore, regular and efficient pipeline damage inspection is crucial.

[0003] Currently, pipeline defect detection faces the following main challenges. Firstly, existing pipeline defect detection methods typically rely on highly complex models such as Transformer and RefineDet. While these methods offer high accuracy, they generally suffer from large parameter counts, high computational costs, and deployment difficulties, making it difficult to meet the urgent industrial demands for real-time performance, low power consumption, and lightweight design. Secondly, the effectiveness of internal pipeline defect detection is limited by the complex physical environment caused by its unstructured space. In this environment, uneven illumination, localized strong reflections, and interwoven shadows are common, while interference from media such as water accumulation, oil stains, and fog further affects image quality. These factors not only significantly reduce the image signal-to-noise ratio but may even completely obscure the area to be inspected, causing systems relying on single visual information to experience partial or complete information loss in critical areas, thus posing a fundamental challenge to reliable defect identification.

[0004] On the other hand, the complex internal environment further increases the difficulty of identification. For example, structural or environmental traces such as welds, rust, and water stains are visually highly similar to real defects (such as cracks and corrosion), generating a large number of false defects and greatly increasing the risk of misjudgment by traditional visual models. In addition, defects themselves exhibit diversity in visual representation. In terms of scale, targets can range from sub-millimeter-level microcracks to large-area corrosion spanning several meters; in terms of morphology, similar defects may present completely different appearances such as dots, patches, or grooves. This huge intra-class and inter-class difference places extremely stringent demands on the model's feature learning and generalization capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent pipeline defect detection method and system. This method can achieve real-time and accurate defect identification and location in resource-constrained industrial environments, effectively overcome interference from complex sensing environments, and efficiently integrate multimodal information to improve detection reliability in the presence of false defects and diverse defects.

[0006] To achieve the above objectives, the present invention provides the following solution: A deep learning-based method for cross-modal pipeline defect detection includes: The RGB and depth images of the pipeline to be detected are obtained, and a dual-branch depth detection network is used to extract features from the RGB and depth images respectively to obtain RGB features and depth features. In the high-level semantic layer of the dual-branch deep detection network, a feature fusion adapter is used to perform multimodal feature alignment and adaptive weighted fusion on the RGB features and the deep features to obtain fused features; Based on the fusion features, the detection head enables the detection of defects in the pipeline.

[0007] Preferably, the dual-branch depth detection network adopts a symmetrical parallel encoder-decoder architecture, including an RGB branch and a depth branch. The two branches are aligned and adaptively weighted fused at the P4 level through a feature fusion adapter.

[0008] Preferably, the method for obtaining the fusion feature includes: Channel alignment and edge enhancement are performed on the depth features to obtain enhanced depth features; Frequency domain separation operations are performed on RGB features and enhanced depth features respectively to extract the low-frequency features of the corresponding RGB mode and the high-frequency features of the depth mode; By utilizing a cross-modal fusion attention mechanism, the low-frequency features of the RGB modality and the high-frequency features of the deep modality are fused across modalities to obtain cross-modal fusion features; The cross-modal fusion features are enhanced using a spatial attention mechanism to obtain multi-scale enhanced features; The quality of RGB and depth features is evaluated by adaptive fusion gating, and three fusion weights for RGB branch, depth branch and cross-modal fusion branch are generated based on the quality evaluation results. Based on the three-way fusion weights, the RGB features, depth features, and multi-scale enhancement features are weighted and summed to obtain the fused features.

[0009] Preferably, the method for frequency domain separation of RGB features and enhanced depth features includes: using a 3×3 depthwise separable convolution to perform mean fuzzing on the RGB features and enhanced depth features to obtain low-frequency features of the RGB mode and deep low-frequency features; obtaining high-frequency features of the depth mode based on the residual between the depth features and the deep low-frequency features; wherein, the low-frequency features of the RGB mode correspond to the overall structural information of the pipeline, and the high-frequency features of the depth mode correspond to the geometric edge information of pipeline defects; low-frequency weights and high-frequency weights are generated by introducing a frequency self-gating mechanism to modulate the low-frequency features of the RGB mode and the high-frequency features of the depth mode.

[0010] Preferably, methods for obtaining cross-modal fusion features include: Lightweight channel dimensionality reduction and feature projection are performed on the low-frequency features of the RGB mode and the high-frequency features of the depth mode to obtain the low-frequency feature sequence of the projected RGB mode and the high-frequency feature sequence of the projected depth mode. By utilizing a bidirectional cross-attention architecture based on a multi-head attention mechanism, bidirectional cross-modal attention is calculated using the low-frequency feature sequence of the projected RGB modality and the high-frequency feature sequence of the projected depth modality. The low-frequency features of the RGB modality after bidirectional cross-modal attention enhancement are fused with learnable weights and combined with lightweight residual connections to obtain the cross-modal fused features.

[0011] Preferred methods for obtaining the three-way fusion weights include: A lightweight quality assessment network is used to score the quality of RGB features and depth features respectively, resulting in RGB quality scores and depth quality scores. Based on the RGB quality score, depth quality score, quality difference, and modal complementarity index, a fusion decision score is generated through a fusion decision network. Based on the RGB quality score, depth quality score, and fusion decision score, and combined with preset basic weights, the initial weights of the RGB branch, depth branch, and cross-modal fusion branch are dynamically generated. The initial weights of the depth branch and the initial weights of the cross-modal fusion branch are modulated with the depth confidence map to obtain spatially adaptive three-way fusion weights; wherein, the depth confidence map is generated by analyzing the edge strength and local variance of the depth features.

[0012] Preferred methods for obtaining multi-scale enhanced features include: Global average pooling and global max pooling are used to extract global and local defect information of cross-modal fusion features, respectively; Based on global and local defect information, a spatial attention map is generated through a 7×7 convolution. A multi-scale boundary enhancement mechanism is introduced for RGB features to obtain enhanced RGB boundary features; The enhanced RGB boundary features are injected into the cross-modal fusion features using a spatial attention map to obtain the final cross-modal fusion features; Multi-scale enhancement of pipeline defect features at three different scales is achieved by using three 3×3 depthwise separable convolutions with different expansion rates. By employing a multi-scale adaptive fusion strategy, we obtain enhanced features at three different scales and the fusion weights of the final cross-modal fusion features, and then perform multi-scale feature fusion to obtain multi-scale enhanced features.

[0013] This invention also provides a deep learning-based cross-modal pipeline defect detection system for implementing the method, comprising: The image acquisition module is used to acquire the RGB image and depth image of the pipeline to be detected, and to use a dual-branch depth detection network to extract features from the RGB image and depth image respectively to obtain RGB features and depth features. The feature fusion module is used to perform multimodal feature alignment and adaptive weighted fusion of the RGB features and the depth features in the high-level semantic layer of the dual-branch deep detection network using a feature fusion adapter to obtain fused features. The defect detection module is used to detect defects in the pipeline based on the fused features and through a detection head.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: By introducing a feature fusion adapter for hierarchical cross-modal fusion, this invention can effectively improve the accuracy and robustness of pipeline defect detection in resource-constrained industrial scenarios. Its advantages are: (1) The hierarchical injection and three-way gating driven by depth confidence significantly reduce the false detection rate and false negative rate in low signal-noise, weak texture, occlusion and reflective areas; (2) The frequency domain separation and self-gating method can retain structural and detailed information at the same time, and can still stably locate defects under complex background and pseudo-defect interference; (3) The bidirectional cross-modal attention combined with spatial and multi-scale adaptive fusion method effectively solves the problem of semantic and scale mismatch between RGB mode and depth mode, and effectively improves the recall ability of multi-size and multi-form defects; (4) The lightweight depth separable convolution, small head number attention and mild modal dropout control computation and power consumption can maintain real-time inference on edge devices.

[0015] In summary, the direct effects of the above-mentioned technical features are that this method maintains real-time performance while reducing the triggering of false defects, improving the visibility of weak defects and overall detection accuracy, and taking into account the computational and energy consumption constraints of engineering deployment. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the cross-modal attention mechanism structure according to an embodiment of the present invention; Figure 2 This is a flowchart of the quality assessment network in an embodiment of the present invention; Figure 3 This is a schematic diagram of the fusion decision network structure according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the weight generation and constraint process in an embodiment of the present invention. Figure 5 This is a flowchart illustrating the modulation and fusion output process according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the feature fusion adapter structure according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a dual-branch lightweight cross-modal adaptive fusion network structure according to an embodiment of the present invention; Figure 8 The image provided is an example of a pipeline dataset from an embodiment of the present invention; wherein, (a) represents Deformation; (b) represents Obstacle; (c) represents Rupture; (d) represents Disconnect; (e) represents Misalignment; and (f) represents Deposition. Figure 9 This is an example of experimental results for the pipeline dataset in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1: like Figure 1 As shown, a deep learning-based method for cross-modal pipeline defect detection includes: S1: Obtain the RGB image and depth image of the pipeline to be detected, and use a dual-branch depth detection network to extract features from the RGB image and depth image respectively to obtain RGB features and depth features; S2: In the high-level semantic layer of the dual-branch deep detection network, a Feature Fusion Adapter (FFA) is used to perform multimodal feature alignment and adaptive weighted fusion of RGB features and depth features to obtain fused features. Specifically, the adapter first performs channel alignment and edge enhancement on the depth features, and estimates the depth confidence using Sobel gradient, Laplacian energy, and local variance. After separating the RGB features and depth features in the frequency domain to obtain low-frequency and high-frequency components, a progressive training strategy is adopted. When the training progress reaches 35%, the P4 depth pyramid compensation branch is activated, generating gating coefficients based on the confidence level and injecting high-frequency depth features. Adaptive weighting of RGB low-frequency and high-frequency depth features is performed through a frequency self-gating mechanism. The weights of the RGB branch, depth branch, and fusion branch are generated using adaptive fusion gating. Bidirectional cross-modal attention and spatial attention enhancement are performed, combined with multi-scale dilated convolution and adaptive scale fusion, to finally output the fused features.

[0021] A further implementation method is that the dual-branch depth detection network adopts a symmetrical parallel encoder-decoder architecture, including an RGB branch and a depth branch. The two branches are aligned and adaptively weighted fused at the P4 level through a feature fusion adapter.

[0022] A further implementation method includes obtaining the fusion features by: S21: Channel alignment and edge enhancement are performed on depth features to obtain enhanced depth features. Specifically, damage such as cracks, corrosion, and dents on the pipe surface manifests as geometric discontinuities in depth images, while in RGB images they may appear as color changes or texture breaks. To fully utilize the complementary information of these two modalities, FFA first aligns and enhances the depth features using a 1×1 convolution. The aligned features are then normalized and activated using batch normalization and the SiLU activation function, ensuring that the distribution of depth features is consistent with that of RGB features in terms of channel dimension and feature quality, thus facilitating subsequent cross-modal interaction. The alignment process is implemented in the following way: (1) in, The number of RGB feature channels represents the dimension of the surface texture features extracted from the RGB image of the pipeline; The depth primitive features represent the geometric structural features extracted from the depth image of the pipe, including geometric information such as the height variation and indentation depth of the pipe surface.

[0023] Subsequently, since various pipeline defects manifest in depth images as abrupt changes in depth values ​​and localized depth anomalies, edge enhancement can more clearly highlight these abnormal areas. For example, cracks on the pipeline surface appear as discontinuous depth along the crack direction in the depth image, while depressions appear as significant reductions in local depth values. Therefore, this embodiment employs a multi-scale depth-separable convolution (3×3, 5×5, 7×7) in FFA to capture edge information at different scales. Specifically, the 3×3 convolution captures small cracks and localized depressions, the 5×5 convolution captures medium-sized corrosion areas, and the 7×7 convolution captures large-area pipeline deformations. Feature stitching is performed through 1×1 convolutions in the fusion layer, integrating multi-scale edge features into a unified enhanced representation, thereby enabling the simultaneous detection of pipeline defects at different scales.

[0024] S22: Perform frequency domain separation operations on the RGB features and enhanced depth features respectively to extract the low-frequency features of the corresponding RGB mode and the high-frequency features of the depth mode; a further implementation method is that the method of performing frequency domain separation on the RGB features and enhanced depth features respectively includes: using a 3×3 depthwise separable convolution to perform mean fuzzing on the RGB features and enhanced depth features to obtain the low-frequency features of the RGB mode and the low-frequency features of the depth mode; based on the residuals of the depth features and the low-frequency features of the depth mode, the high-frequency features of the depth mode are obtained; wherein, the low-frequency features of the RGB mode correspond to the overall structural information of the pipeline, and the high-frequency features of the depth mode correspond to the geometric edge information of the pipeline defects, and low-frequency weights and high-frequency weights are generated by introducing a frequency self-gating mechanism to modulate the low-frequency features of the RGB mode and the high-frequency features of the depth mode.

[0025] Specifically, in pipeline defect detection, different frequency components carry different damage information. Low-frequency components of RGB images mainly contain information about the overall shape and structure of the pipeline, such as the pipeline outline and large areas of corrosion; while high-frequency components mainly contain texture and detail information, such as fine crack textures and surface roughness variations. Low-frequency components of depth images mainly contain the overall geometric structure of the pipeline, such as the basic shape and overall deformation; while high-frequency components mainly contain edges and geometric discontinuities, such as abrupt depth changes at cracks and geometric discontinuities at recessed edges. To fully utilize this frequency domain complementarity, this embodiment employs a frequency domain separation strategy, decomposing RGB and depth features into low-frequency and high-frequency components, respectively. Specifically, a 3×3 depthwise separable convolution is used to perform mean blurring on the features to obtain the low-frequency components, as shown below: (2) in, The input features are the original features. For RGB features, these represent the texture and color information of the pipe surface; for depth features, they represent the geometric height information of the pipe surface. The Blur operation is implemented through depthwise separable convolution. Each channel undergoes an independent 3×3 convolution with a uniformly distributed kernel weight of 1 / 9 to achieve a mean blurring effect, thereby extracting the overall structural information of the pipe. Furthermore, the high-frequency components are the residuals between the original features and the low-frequency components. For RGB features, the high-frequency components contain fine crack textures and surface details; for depth features, the high-frequency components contain geometric discontinuities and edge information of the damaged area.

[0026] During frequency domain fusion, the most complementary frequency components are extracted from RGB and depth features respectively. Since the low-frequency components of the RGB image primarily carry information about the overall pipe structure and contour, such as the overall shape of large-area corrosion regions, the FFA module mainly extracts the low-frequency components of the RGB features to provide overall contextual information about pipe damage. The high-frequency components of the depth image, on the other hand, are concentrated on geometric edges and deformation details, such as abrupt depth changes at cracks and geometric discontinuities at recessed edges; therefore, the high-frequency components of the depth features are extracted. This process aims to enable the cross-modal attention mechanism to interact based on complementary frequency domain characteristics, thereby effectively fusing the structural context (overall damaged area) of the RGB modality with the geometric edge information (damage boundaries and geometric discontinuities) of the depth modality, enhancing the ability of multimodal feature representations to discriminate pipe damage.

[0027] Based on the feature quality of RGB low-frequency and depth high-frequency signals, a frequency self-gating mechanism (FG) is introduced in the FFA module to generate low-frequency weights to suppress noisy high-frequency signals or overly smoothed low-frequency signals. In pipe damage detection, when the depth image quality is poor (e.g., uneven illumination, surface reflection), the depth high-frequency signals may contain a lot of noise, requiring a reduction in their weights. Conversely, when the RGB image is blurry or has low contrast, the RGB low-frequency signals may be overly smoothed, necessitating an enhancement of the depth high-frequency contribution. The gating weights are primarily generated through global average pooling and a two-layer fully connected network. This design allows the model to dynamically adjust the frequency domain fusion strategy based on the quality of the input features, avoiding noise interference or information loss, while retaining at least 30% of the original features to prevent over-suppression leading to missed detections. Details are as follows: (3) in, The channel dimension of the low-frequency features of the RGB modality and the high-frequency features of the deep modality is concatenated; GAP represents global average pooling; MLP is a two-layer fully connected network multilayer perceptron used to learn the mapping relationship between feature quality and fusion weights. Use the Sigmoid activation function. Low-frequency weights. Used to modulate RGB low frequency and deep high frequency, high frequency weighting Used to modulate deep high frequencies, thereby adaptively fusing pipeline damage information based on feature quality.

[0028] Furthermore, to ensure the stability of the depth high-frequency signal in FFA under noisy conditions such as pipe surface reflection and uneven illumination, this embodiment employs a progressive training strategy. When the training progress reaches 35%, the P4 depth pyramid compensation branch is activated. This branch uses a 3×3 convolution with a stride of 2 for downsampling, followed by upsampling using a transposed convolution, thereby generating stable P4 depth features—multi-scale depth information—capturing pipe damage at different scales. This feature is modulated using a sigmoid gating mechanism driven by depth channel confidence, generating gating coefficients based on the confidence level and injecting them into the depth high frequency. The depth confidence reflects the reliability of the depth information in pipe damage detection. For example, in high-confidence regions, such as clear crack edges, the depth information is considered reliable, and its weight should be increased; in low-confidence regions, such as reflective areas, the depth information is considered unreliable, and its weight should be decreased.

[0029] To further alleviate the unreliability of depth information under low confidence levels, a high-frequency RGB feature compensation term is introduced, and a hierarchical fusion strategy is adopted: 1. In low-confidence regions (<0.3), depth information is unreliable, such as in reflection areas. RGB high-frequency data (crack texture) will be fused with a weight of 0.12 to compensate for the lack of depth information. 2. In the medium confidence region (0.3-0.5), the depth information is partially reliable. RGB high frequencies will be fused with a weight of 0.08. 3. In high-confidence regions (≥0.5), depth information is reliable, such as clear damage edges. RGB high frequencies will be fused with a weight of 0.05.

[0030] This spatially adaptive fusion strategy can dynamically adjust the fusion strength according to the confidence level of each location, effectively improving the robustness and reliability of deep features in noisy environments and ensuring the accuracy of pipeline damage detection.

[0031] S23: Utilizing a cross-modal fusion attention mechanism, low-frequency features of the RGB modality and high-frequency features of the deep modality are fused across modalities to obtain cross-modal fusion features. A further implementation method for obtaining cross-modal fusion features includes: performing lightweight channel dimensionality reduction and feature projection on the low-frequency features of the RGB modality and the high-frequency features of the deep modality to obtain projected low-frequency feature sequences of the RGB modality and projected high-frequency feature sequences of the deep modality; using a bidirectional cross-attention architecture based on a multi-head attention mechanism, calculating bidirectional cross-modal attention using the projected low-frequency feature sequences of the RGB modality and the projected high-frequency feature sequences of the deep modality; fusing learnable weights on the low-frequency features of the RGB modality and the high-frequency features of the deep modality enhanced by bidirectional cross-modal attention, and combining this with lightweight residual connections to obtain cross-modal fusion features.

[0032] Specifically, in pipeline defect detection tasks, RGB images and depth images provide complementary defect information. RGB images contain rich texture, color, and surface detail information, such as crack texture features, color variations in corrosion areas, and surface roughness; depth images provide geometric structure, spatial relationships, and shape information, such as depth discontinuities at cracks, decreased depth in concave areas, and increased depth in convex areas. However, traditional feature fusion methods (such as simple stitching, weighted averaging, or element-wise addition) cannot establish deep cross-modal dependencies and struggle to fully utilize the complementarity of the two modes. Cracks on the pipeline surface may appear as darker colors or texture discontinuities in RGB images, while in depth images they may appear as abrupt changes in depth values, forming geometric edges. Furthermore, structural deformations on the pipeline surface (such as depressions and convexities) are more easily identified in depth images, manifesting as significant changes in local depth values, but may only appear as slight shadows or color variations in RGB images. Therefore, this embodiment designs a cross-modal attention mechanism (CMA) in the FFA module, which can establish long-distance dependencies across modalities, enabling RGB features to focus on geometric information in depth features, while depth features can also focus on texture information in RGB features, such as... Figure 1 As shown. For example, crack textures in RGB features can focus on the corresponding geometric discontinuities in depth features, and geometric edges in depth features can focus on the corresponding texture breaks in RGB features, thereby achieving accurate identification of pipeline defects.

[0033] Lightweight channel dimensionality reduction and feature projection: To address the computational overhead caused by high-dimensional feature maps in cross-modal fusion, this invention employs the following specific implementation method: CMA first performs lightweight channel dimensionality reduction on RGB and depth features. By reducing feature dimensions, computational overhead is reduced while preserving complete spatial information, enabling the model to perform cross-modal interactions at the original resolution, thereby accurately locating pipeline defects. However, an excessively small dimensionality reduction ratio significantly reduces feature expressiveness, leading to loss of defect information, such as the compression and loss of detailed crack features. Conversely, an excessively large dimensionality reduction ratio fails to effectively reduce the model's computational overhead. Therefore, we select a trade-off channel dimensionality reduction ratio that effectively reduces the computational complexity of attention while maintaining a certain level of feature expressiveness.

[0034] Feature dimensionality reduction is a commonly used data processing technique in machine learning and deep learning. This technique maps high-dimensional feature data to a low-dimensional space using specific algorithms, reducing the number of features while preserving as much key information as possible from the original data. This reduces computational complexity and improves the model's generalization ability. The dimensionality of the reduced features typically needs to be divisible by the number of attention heads to ensure the correct implementation of subsequent multi-head attention mechanisms. After determining the dimensionality for dimensionality reduction, RGB features and depth features are projected through independent 1×1 convolutional layers. This operation allows features at each spatial location to undergo independent linear transformations along the channel dimension, thus achieving dimensionality reduction while preserving the pipeline defect information corresponding to each pixel location and effectively reducing the number of model parameters.

[0035] In contrast, using a fully connected layer as the projection layer requires flattening the feature map into a one-dimensional vector. This process not only disrupts the original spatial structure but also introduces higher computational complexity. Furthermore, batch normalization incurs additional computational and memory overhead, while activation functions introduce non-linearity. Introducing batch normalization and activation functions into the projection layer may interfere with the attention mechanism's direct modeling of cross-modal associations, affecting the association learning between RGB texture and depth geometry information, and further increasing the computational burden.

[0036] Since the projected features are two-dimensional feature maps, they need to be converted into a sequence. Specifically, the feature vector of each spatial location is regarded as an element in the sequence, and each element corresponds to a spatial location on the pipe surface, containing the defect information at that location. Subsequently, the spatial dimension is converted into sequence length through a flattening operation to obtain the feature representation in sequence form, thereby enabling the attention mechanism to establish the correlation between different spatial locations, such as the correlation between different parts of a crack.

[0037] Bidirectional cross-modal attention computation: After feature channel dimensionality reduction and feature projection, the cross-modal attention mechanism performs correlation modeling on the low-frequency features of the projected RGB modality and the high-frequency features of the depth modality. This process calculates the attention weight matrix between modalities to achieve dynamic focusing of key features and suppression of redundant information, thus providing accurate feature input for subsequent hierarchical fusion. Specifically, unlike traditional self-attention mechanisms, CMA first adopts a bidirectional cross-attention architecture. This architecture constructs two independent attention paths, one for queries of RGB features and key-value pairs of depth features, and the other for queries of depth features and key-value pairs of RGB features. This design allows each modality to extract relevant information from the other modality, achieving effective cross-modal information exchange. For example, crack textures in RGB features can extract corresponding geometric discontinuities from depth features, and geometric edges in depth features can extract corresponding texture fracture information from RGB features. This is specifically achieved in the following ways: (4) (5) in, The parameters are query, key, and value. and These are the flattened RGB and depth feature sequences, respectively.

[0038] Secondly, to enhance the model's ability to capture cross-modal relationships, a multi-head attention (MHA) mechanism is introduced into the bidirectional cross-attention architecture, forming a bidirectional multi-head cross-attention (BMCA) architecture. This mechanism first divides the dimensionality-reduced features into multiple attention heads along the channel dimension. Each head is computed in parallel in an independent subspace to focus on different aspects of cross-modal correlations. For example, different attention heads can focus on different types of pipeline defects. Subsequently, the outputs of all attention heads are concatenated and fused. This process effectively enriches the diversity of the final synthesized features and the overall representational capability of the model. In RGB-to-depth attention, RGB features serve as queries, and depth features serve as keys and values. This means that RGB features will focus on geometric information related to themselves in the depth features; for example, crack textures in RGB will focus on the corresponding geometric discontinuities in the depth. In depth-to-RGB attention, depth features serve as queries, and RGB features serve as keys and values. This means that depth features will focus on texture information related to themselves in the RGB features; for example, geometric edges in the depth will focus on the corresponding texture breaks in the RGB.

[0039] Weight calculation and feature fusion: Each attention head independently generates an attention weight by calculating the similarity between the query and the key; this weight is then multiplied by the value matrix to obtain a weighted feature representation. In pipeline defect detection scenarios, different spatial parts of a crack exhibit high correlation, while the correlation between the defective region and the surrounding normal region is significantly lower. This attention weight accurately captures and reflects the degree of correlation between different spatial locations in the image, thus effectively highlighting the feature information of the defective region. The specific representation is shown below: , (6) in, , , These are query, key, and value matrices, all with the following shape: The dimensions of each head are: Attention weights are calculated based on the similarity between the query and the key, specifically as follows: Matrix multiplication yields a matrix of shape... The similarity matrix, where each element Indicates the first The spatial location is the first Attention level at each spatial location; scaling factor This is used to prevent the gradient from vanishing due to excessively high attention scores. The function normalizes the similarity matrix so that the sum of the attention weights for each query position to all key positions is 1, thus forming a probability distribution representing the attention to the defect region for each position. Finally, the attention weights are multiplied by the value matrix to obtain the weighted feature representation, with the shape [formula missing]. This means that the features at each location incorporate defect information from other locations. Furthermore, the outputs of multiple attention heads are concatenated and then restored to their original dimensions through an output projection layer, resulting in a shape... That is, each location contains fused defect information.

[0040] To stabilize training and preserve key information, CMA introduces residual connections and layer normalization after the attention layer. Residual connections enable the model to learn cross-modal augmented residual information, preventing the loss of crucial modal information. Layer normalization standardizes the features of each sample, making the feature distribution more consistent across different samples, which helps improve training stability. This design allows attention-enhanced information to be smoothly integrated into the original features, avoiding drastic feature changes that could lead to training instability and ensuring the effectiveness of pipeline defect detection.

[0041] Cross-modal fusion: To remap the defect features, which integrate RGB texture and depth geometry information, back to the original feature space, this embodiment uses a 1×1 convolutional layer as the projection layer. By restoring the original number of channels, it ensures compatibility with the input dimension of subsequent networks. The restored feature map undergoes transpose and reshaping operations to restore its spatial structure and maintain the original spatial resolution. That is, each pixel position corresponds to a specific location on the pipe surface, containing the defect information at that location.

[0042] Subsequently, the RGB attention features enhanced with deep geometric information and the deep attention features enhanced with RGB texture information are adaptively fused using learnable weights. The fusion weights are generated from learnable parameters normalized using Softmax. The initial value of the learnable parameters is set to 0.5, meaning that in the initial state, the two modalities contribute equally to defect detection. Softmax normalization ensures that the sum of the weights is 1, representing the degree of dependence of defect detection on the two modalities, thus giving the fusion process probabilistic interpretability. Furthermore, the final fused features are compressed using a 1×1 convolution to achieve deep information integration.

[0043] To preserve the original defect texture information in the RGB image, a lightweight residual connection is added to the final output of CMA. By retaining some original RGB features, this residual structure enhances the model's robustness and avoids over-reliance on attention features. For example, when the attention mechanism fails, defect detection can still rely on original RGB features. On the one hand, excessively large coefficients weaken the effect of the attention mechanism and reduce the pipeline detection performance of cross-modal fusion. On the other hand, when the coefficients are too small, resulting in unreliable depth information, it is impossible to rely on RGB features and retain the original information. After comprehensive evaluation, a base residual coefficient of 0.25 is chosen, achieving an optimal balance between maintaining cross-modal fusion performance and preserving sufficient original RGB defect information. Furthermore, in low-confidence scenarios, the residual coefficient dynamically increases to approximately 0.33, further enhancing the preservation of original information and thus relying more on RGB features for defect detection.

[0044] S24: Enhance cross-modal fusion features using a spatial attention mechanism to obtain multi-scale enhanced features; a further implementation method includes: Global average pooling and global max pooling are used to extract global and local defect information from the cross-modal fusion features, respectively. Based on the global and local defect information, a spatial attention map is generated through a 7×7 convolution. A multi-scale boundary enhancement mechanism is introduced for the RGB features to obtain enhanced RGB boundary features. The enhanced RGB boundary features are then injected into the cross-modal fusion features through the spatial attention map to obtain the final cross-modal fusion features. Specifically, the low-frequency features of the RGB modality (corresponding to the overall structural information of the pipeline) and the high-frequency features of the depth modality (corresponding to the geometric edge information of defects) are deeply interacted through the CMA module to generate defect fusion features that fuse RGB texture details and depth geometric features. This mechanism enhances complementarity through bidirectional cross-modal attention: RGB texture features focus on depth geometric information, while depth geometric features are associated with RGB texture details, effectively integrating the advantages of both modalities to improve the representational ability of defect features.

[0045] Subsequently, the fused features are further enhanced using a spatial attention mechanism. Specifically, global average pooling and global max pooling are employed for fusion to extract global and local defect information. After concatenating the two along the channel direction, a spatial attention map is generated using a 7×7 convolution and a sigmoid activation function to identify important defect regions. The spatial attention map is used to modulate cross-modal attention features, highlighting the spatial regions of defects and suppressing background regions to obtain spatially enhanced features.

[0046] To further enrich the details, a multi-scale boundary enhancement mechanism is introduced into the original RGB features to highlight texture and edge information in the RGB image, such as crack textures and erosion edges. Subsequently, the enhanced RGB boundary features are injected into the cross-modal features under the guidance of spatial attention, resulting in the final cross-modal features. This design allows RGB boundary information to be selectively injected into the cross-modal features under the guidance of spatial attention—that is, injected only into defect areas and not into background areas—thereby further enhancing the texture and edge information in defect areas.

[0047] Multi-scale enhancement of pipeline defect features at three different scales is achieved by using three 3×3 depthwise separable convolutions with different expansion rates. Enhanced features at three different scales are obtained by using a multi-scale adaptive fusion strategy to obtain the enhanced features at three different scales and the fusion weights of the final cross-modal fusion features. Multi-scale feature fusion is then performed to obtain multi-scale enhanced features.

[0048] Specifically, since micro-cracks require fine-grained features to capture subtle texture details, and large-area corrosion requires large receptive field features to capture extensive structural changes, this embodiment employs a multi-scale feature enhancement strategy to address the multi-scale characteristics of pipeline defects. This strategy uses dilated convolutions with different dilation rates to enhance features at multiple scales. Specifically, the features are processed using three 3×3 depthwise separable convolutions with different dilation rates (1, 2, and 3, respectively). The details are shown below: (7) in, This indicates a 3×3 depthwise separable convolution used to extract defect features at different scales; This indicates batch normalization, which standardizes the distribution of features. The SiLU activation function is used to activate nonlinear features. Features at each scale retain the original spatial resolution but have different receptive fields. . It can detect minute cracks and local defects by focusing on local details; It corresponds to a medium-sized structure and detects corrosion areas within a medium range. It can detect large-scale pipeline deformation and overall structural changes in a wide range of contexts.

[0049] Furthermore, the multi-scale features include cross-modal features and enhancement features at three different scales, totaling four feature maps. To fully utilize these multi-scale features, this embodiment employs a multi-scale adaptive fusion strategy, dynamically adjusting the fusion weights based on the feature at each scale of the defect. This strategy is primarily implemented through global average pooling and a lightweight convolutional network. Details are as follows: (8) in, It represents the original cross-modal features, containing defect information at all scales; The enhancement features are categorized into three different scales, corresponding to local, medium, and large-scale defect features, respectively. For example, when detecting micro-cracks, the weight of local-scale features will increase; when detecting large-area corrosion, the weight of large-scale features will increase.

[0050] Finally, the features from all scales are weighted, concatenated, and fused using a 1×1 convolution. This fusion is then added to the cross-modal features to output multi-scale enhanced features. The details are shown below: (9) S25: Perform quality assessment on RGB features and depth features through adaptive fusion gating, and generate three fusion weights for RGB branch, depth branch and cross-modal fusion branch based on the quality assessment results.

[0051] In pipeline defect detection scenarios, the quality of RGB images and depth images often differs significantly. RGB images typically have a high signal-to-noise ratio and stable texture information, clearly displaying crack textures and color changes in corrosion areas. However, depth images are susceptible to factors such as lighting conditions, surface reflection characteristics, and sensor accuracy, leading to unstable image quality. For example, depth information in strongly lit areas may be inaccurate, and depth values ​​in reflected areas may be abnormal. Traditional fixed-weight fusion strategies cannot adapt to these dynamic changes, potentially introducing low-quality depth information that interferes with RGB features, resulting in reduced overall detection performance. Therefore, this embodiment designs an Adaptive FusionGate (AFG) to address the modal quality difference problem in multimodal feature fusion. It dynamically adjusts the fusion strategy based on the quality of the RGB and depth images, ensuring full utilization of geometric information when depth information is reliable, and relying more on RGB texture information when depth information is unreliable.

[0052] A further implementation method involves obtaining the three-way fusion weights by: A lightweight quality assessment network is used to score the quality of RGB features and depth features separately, resulting in RGB quality scores and depth quality scores. Based on the RGB quality scores, depth quality scores, quality differences, and modal complementarity indices, a fusion decision network is used to generate a fusion decision score. Specifically, firstly, AFG uses two independent lightweight quality assessment networks to evaluate the quality of RGB features and depth features separately, such as... Figure 2 As shown, each quality assessment network uses Global Average Pooling (GAP) to compress spatial features into a global descriptor, extracting feature quality information from the entire pipe surface. Subsequently, a two-layer fully connected network maps the global descriptor to a single quality score. This quality score is normalized to the [0,1] interval using a Sigmoid activation function, representing the reliability of the corresponding modal feature.

[0053] During training, quality assessment networks can learn criteria to distinguish between high-quality and low-quality features. For example, for RGB features, the network may learn metrics such as texture richness and edge sharpness; for depth features, the network may learn metrics such as geometric consistency and noise level. In pipeline defect detection, high-quality RGB features can clearly show the color changes of crack textures and corrosion areas, while high-quality depth features can clearly show the geometric discontinuities of defect areas.

[0054] Based on the quality assessment results, this embodiment constructs a fusion decision network, specifically as follows: Figure 3As shown, this network comprehensively considers four key factors: RGB quality score, depth quality score, quality difference, and complementarity index. Quality difference reflects the degree of reliability difference between the two modalities. A large quality difference indicates that one modality is significantly superior to the other, and more reliance should be placed on the higher-quality modality. A small quality difference indicates that the two modalities are of comparable quality, allowing for more balanced fusion. The complementarity index measures both modalities and also provides the degree of reliable information. High complementarity indicates that both modalities provide valuable information, and cross-modal fusion features receive higher weights. The fusion decision network concatenates these four indices and inputs them into a two-layer fully connected network to generate a fusion decision score, which determines how to fuse RGB, depth, and cross-modal features for defect detection.

[0055] Based on the RGB quality score, depth quality score, and fusion decision score, and combined with preset base weights, initial weights for the RGB branch, depth branch, and cross-modal fusion branch are dynamically generated. Specifically, this embodiment employs an adaptive weight generation strategy based on quality scores to generate independent fusion weights for RGB features, depth features, and cross-modal fusion features, as detailed below. Figure 4 As shown. Unlike traditional fixed-weight or simple attention mechanisms, AFG's weight generation strategy combines prior knowledge and data-driven quality assessment. Based on extensive experimental analysis, we designed basic weights of 0.3, 0.4, and 0.32 to ensure a reasonable fusion ratio even when quality assessment is uncertain. Furthermore, this embodiment also designs three sets of dynamically adjusted quality weights of 0.25, 0.3, and 0.45 for actual quality conditions. These weights combine the RGB quality score, depth quality score, and fusion quality score to perform weighted fusion of the basic weights. For example, when the depth image quality is high, the depth feature weight increases; when the RGB image quality is high, the RGB feature weight increases; and when both modalities are reliable, the cross-modal fusion feature weight increases.

[0056] The initial weights of the depth branch and the cross-modal fusion branch are modulated with the depth confidence map to obtain spatially adaptive three-way fusion weights. The depth confidence map is generated by analyzing the edge strength and local variance of depth features. Specifically, reliable depth information typically has clear geometric edges and moderate local variations, while unreliable depth information often exhibits blurred edges, excessive noise, or abnormal local variations. To further improve the robustness of the fusion, AFG modulates the generated depth weights and fusion weights with the depth confidence map (DCM), as follows: Figure 4As shown, the depth confidence map is generated by analyzing the edge strength and local variance of depth features, reflecting the reliability of depth information for defect detection at each spatial location. Specifically, the DCM calculates the edge strength of depth features using the Sobel gradient operator to detect the geometric edges of defect regions, such as depth discontinuities at crack edges; it combines the second derivative information captured by the Laplacian operator to detect geometric discontinuities in defect regions, such as depth variations in recessed areas; and it estimates the stability of detected depth values ​​through local variance, comprehensively evaluating the reliability of depth information at each spatial location, such as depth value fluctuations in noisy regions. The details are shown below: (10) in, Represented as a depth feature map; and These are represented as the Sobel gradients in the horizontal and vertical directions, respectively; Represented as the Laplace operator; Represented as local variance; It is a small constant to prevent division by zero.

[0057] Furthermore, AFG employs linear modulation for the depth weights and square-root modulation for the fusion weights. This design makes the fusion weights more sensitive to changes in depth confidence. When depth confidence decreases, the fusion weights decrease more rapidly, relying more on RGB features. However, because the fusion features combine RGB and depth information through a cross-modal attention mechanism, even if the depth information is unreliable in a certain region, the fusion features may still obtain compensation from the RGB feature information.

[0058] The quality scores and weights generated by AFG are initially global scalars, representing the overall image quality. To support spatially adaptive fusion, these weights are expanded to the same spatial dimension as the feature maps. During this expansion, the weights are stabilized through extremely lightweight spatial smoothing to prevent excessive differences in fused weights between adjacent pixels, which could lead to discontinuities in the detection results. This smoothing operation ensures the spatial continuity of the weights, avoiding feature discontinuities caused by abrupt changes in spatial weights, which is crucial for maintaining the smoothness and consistency of the detection results. Finally, the three feature paths are weighted and fused using adaptive weights.

[0059] This adaptive fusion gating enables the model to dynamically adjust the fusion strategy based on the modal quality of the input samples, fully utilizing high-quality depth information while preserving RGB performance. Specifically, for example... Figure 5As shown, in pipeline defect detection tasks, when the depth image quality is high (e.g., good lighting conditions and low surface reflection), the model increases the weights of depth features and fused features to fully utilize geometric information; when the depth image quality is low (e.g., areas with strong lighting and high reflection), the model automatically reduces the depth-related weights and relies more on RGB features, achieving more robust and accurate multimodal feature fusion.

[0060] S26: Based on three-way fusion weights, the RGB features, depth features, and multi-scale enhanced features are weighted and summed to obtain the fused features. Specifically, the features after multi-scale enhancement need to be finally fused with the original RGB features and depth features. The three-way fusion weights generated by FFA through AFG correspond to the multi-scale enhanced features, the original RGB features, and the depth features, respectively. The weighted summation method realizes the adaptive fusion of cross-modal features, effectively balancing the contribution of different modal information, providing a more robust fused feature representation for subsequent pipeline defect detection tasks, and enabling the simultaneous detection of pipeline defects of different scales and types.

[0061] To further improve training stability, FFA employs a weight normalization strategy. Weight normalization is achieved through the Softmax function; the modulated weights are normalized using Softmax to obtain the final three-way fusion weights. This three-way fusion approach allows the model to dynamically adjust the contribution of different features based on the modal quality and feature importance of the input samples, thereby achieving adaptive multimodal feature fusion.

[0062] Furthermore, FFA adds lightweight residual connections to the final output. Excessively large coefficients weaken the effect of fused features and reduce the effectiveness of multimodal fusion; excessively small coefficients fail to effectively preserve original information, meaning that when fused features fail, the original RGB features cannot be relied upon. Therefore, this embodiment selects original RGB features with a base weight of 0.15 to offset the risk of missing depth information. In low-confidence scenarios, the residual coefficients dynamically increase to 0.18, further enhancing the preservation of original information. If the number of output channels differs from the number of RGB channels, the number of output channels needs to be adjusted. In this case, FFA projects the features to the target dimension through an output projection layer consisting of 1×1 convolutions, batch normalization, and SiLU activation functions, ensuring that FFA can be flexibly embedded into different backbone networks. The specific process of the Feature Fusion Adapter (FFA) is as follows: Figure 6 As shown.

[0063] It should be noted that, addressing the limitation of single-modal vision systems in complex pipeline environments, this invention draws upon the design concepts of current advanced lightweight single-branch object detection networks, such as YOLO and RTDETR, to design a hierarchical cross-modal fusion-based dual-branch depth detection network, specifically as follows: Figure 7 As shown, this network constructs a parallel feature extraction pathway for RGB and deep modalities, achieving effective complementarity of cross-modal information through a hierarchical selective fusion strategy. Unlike traditional fusion methods, this network adopts a mid-stage fusion approach, using an adaptive fusion strategy with an FFA module at the high-level semantic layer, while maintaining the integrity of RGB and deep features at the low-level detail layers, achieving a good balance between computational efficiency and detection accuracy.

[0064] In its implementation, the network employs a symmetrical parallel encoder-decoder architecture, with each branch containing a complete feature pyramid extraction structure. The RGB branch constructs a multi-scale feature representation ranging from 64 to 1024 channels through 11 levels; the depth branch ensures the integrity of auxiliary deep modal features through 11 symmetrical levels. Regarding the fusion strategy, the two branches perform feature alignment and adaptive weighted fusion at level P4 using an FFA module, effectively addressing the issue of feature distribution differences between modalities.

[0065] Furthermore, an improved four-scale detection architecture is employed in the feature aggregation section. Based on the standard top-down FPN path, a P2 detection branch is extended, which upsamples high-level semantic information to the lowest-level features. Subsequently, feature re-fusion and refinement are performed through a bottom-up PAN path. Finally, detection results are output in parallel at four different resolutions, effectively improving the detection capability for defects of different sizes.

[0066] Compared to traditional single-branch or early / late-stage fusion architectures, this network offers several technical advantages. The dual-branch structure effectively maintains the independence of each modality's features, avoiding feature confusion caused by premature fusion; the hierarchical fusion strategy achieves progressive information integration from local details to high-level semantics; and the employed FFA module achieves precise and efficient feature complementarity between modalities through adaptive alignment of channels and spatial dimensions. Furthermore, four scale detection heads ensure comprehensive coverage from local details to global context. Through this refined hierarchical processing and intelligent fusion mechanism, the entire network achieves superior detection performance in complex scenarios such as pipeline defect detection.

[0067] S3: Based on fusion features, the detection head enables the detection of defects in pipelines.

[0068] Example 2: This embodiment provides an experimental procedure: 1. Experimental setup.

[0069] 1.1 Experimental environment.

[0070] The detailed configuration specifications of the experimental environment are shown in Table 1. The hardware configuration includes an Intel(R) Core(TM) i9-10900K CPU, an NVIDIA RTX 3090 GPU, and 24GB of memory; the software environment is based on the Windows 10 operating system, uses the PyTorch 2.0.0 deep learning framework, and is equipped with CUDA 12.6 for accelerated computing.

[0071] Table 1 1.2 Experimental training indicators.

[0072] In this embodiment, the training parameters of the model were carefully configured to obtain optimal performance in the experiment. Table 2 summarizes the specific training parameters of the experiment.

[0073] Table 2 To evaluate the effectiveness of the model, we used a variety of evaluation metrics, including the average class accuracy (mAP50) with an intersection-over-union (IU) threshold of 0.5, the average class accuracy (mAP50–90) with IU thresholds of 0.5–0.9, and the number of parameters.

[0074] 2. Dataset.

[0075] Currently, the field of pipeline defect detection lacks large-scale multimodal detection datasets. Training existing models often relies on small samples or single-modal data, resulting in insufficient generalization ability and difficulty adapting to complex and ever-changing real-world pipeline inspection scenarios. Existing public datasets generally suffer from limited sample sizes, incomplete defect type coverage, and limited modal information (e.g., containing only visual images while lacking depth or infrared data). These limitations hinder the effective training and performance verification of cross-modal fusion models, severely restricting the development of pipeline defect detection technology towards intelligence and precision.

[0076] To address the shortcomings of existing datasets, this embodiment combines a real open-source pipeline dataset (from https: / / www.kaggle.com / datasets / simplexitypipeline / pipeline-defect-dataset) with experimental simulations to construct an RGB-D dataset for the Pipeline Defect Dataset, encompassing various typical defects such as deformation, sediment, pipe segment detachment, misalignment, obstacles, and fractures. The experimental simulations in this embodiment utilize the Depth Anything V2 monocular depth estimation model.

[0077] The RGB-D dataset is partitioned in a 7:2:1 ratio and includes 22,120 valid samples. Each sample is simultaneously registered with both RGB visual image and depth information, effectively addressing the shortcomings of existing public datasets, such as limited sample size, incomplete defect coverage, and limited modal information. This provides high-quality data support for the training and validation of cross-modal fusion models. (Images and detailed information of the PipelineDefect Dataset are shown below.) Figure 8 As shown in Table 3. Figure 8 The image provided is an example of a pipeline dataset from an embodiment of the present invention; wherein, (a) represents Deformation; (b) represents Obstacle; (c) represents Rupture; (d) represents Disconnect; (e) represents Misalignment; and (f) represents Deposition. Figure 9 This is an example of experimental results for the pipeline dataset in an embodiment of the present invention.

[0078] Table 3 3. Ablation experiment.

[0079] To systematically evaluate the effectiveness of the proposed model, this embodiment designs a series of ablation experiments based on the Pipeline Defect Dataset, focusing on the role of the feature fusion adapter in the dual-branch structure. Details are shown in Table 4.

[0080] The experimental design encompasses five scenarios. First, an early fusion strategy is employed, directly concatenating RGB and depth features at the input layer to analyze the interference effect and information redundancy of premature fusion on modal features. Second, only a mid-term fusion strategy is used to verify the limitations of simple fusion. Third, a late-term fusion strategy is employed to analyze the impact of high-level semantic interaction alone on fine-grained information. Fourth, the feature fusion adapter is used only in the P5 branch to analyze its fusion effect. Fifth, the feature fusion adapter is used in both the P4 and P5 branches to analyze the cumulative effect of multi-layer fusion by the feature fusion adapter.

[0081] All experiments used the same training parameters and evaluation metrics. The ablation experiment results will reveal the key role of the feature fusion adapter in balancing modal complementarity and computational efficiency by comparing the model performance under different configurations, and provide experimental basis for the subsequent optimization of cross-modal fusion networks.

[0082] Table 4 First, Scheme 1, without introducing any adaptive adjustment mechanism between modalities, directly concatenates the original RGB features and depth features at the input stage. This may lead to the superposition of noise components from different modalities, while ignoring the complementarity and differences of features in the early stages, thus affecting the effectiveness of subsequent feature extraction and the detection accuracy of the model. Second, Scheme 2 directly concatenates features at layers p4 and p5. Although this avoids interference from early fusion, it cannot adaptively adjust the contribution weights of different modalities, is insensitive to differences in modal quality, and cannot effectively suppress noise from low-quality modalities. Scheme 3 adopts completely independent processing of two paths, extracting high-level semantic features separately before fusion, avoiding feature interference. However, it only fuses at the detection result level, lacking fine-grained interaction at the feature level, and cannot fully utilize multimodal complementary information for effective detection. Therefore, compared to the early and late fusion strategies, the mid-term fusion strategy effectively improves detection accuracy. This indicates that the method of fusing after feature extraction is significantly effective for defect detection, but a more intelligent fusion mechanism is still needed.

[0083] Scheme 4 introduces a feature fusion adapter only at layer p5. Compared to the previous three schemes, its fusion effect at layer p5 is good, but the lack of mid-level feature fusion at layer p4 limits the full utilization of multi-scale features. However, the detection performance of Scheme 5 is actually lower. Therefore, if feature fusion adapters are used in both layers p4 and p5, it may lead to over-fusion of features, resulting in loss of modality specificity; in addition, the computational cost and number of parameters will increase significantly, while the marginal benefit of performance improvement will show a decreasing trend.

[0084] 4. Comparison of detection performance with other methods.

[0085] To comprehensively evaluate the performance of the dual-branch lightweight cross-modal adaptive fusion network in detecting various defects in the complex environment inside pipelines, we compared and analyzed the network with several existing mainstream target detection methods. The detection results are shown in Table 5.

[0086] Table 5 Experimental results show that although the single-modal RGB method has made significant progress in the YOLO series, it still has a significant bottleneck in detection accuracy, reaching a maximum of only 95.7%, which is insufficient to meet the requirements for accurate identification of six types of defects in pipeline defect detection, including deformation, obstacles, ruptures, breaks, misalignments, and deposits. In contrast, the multimodal RGB+depth fusion method fully utilizes the texture and color information of RGB images and the spatial geometric information of depth images, demonstrating a significant advantage in fine-grained localization capabilities. Among them, the M2D-LIF method, although having as many as 36.54M parameters, can achieve a detection accuracy of 95.9% and a fine-grained localization accuracy of 89.3%, proving the effectiveness of multimodal fusion in improving detection accuracy. However, its large number of parameters limits its practical deployment on edge devices.

[0087] The feature fusion adapter proposed in this embodiment maintains a lightweight design while achieving optimal overall performance on a dual-branch lightweight cross-modal adaptive fusion network. Specifically, it achieves a detection accuracy of 96.0%, surpassing other methods. It achieves a fine-grained localization accuracy of 84.1%, which, although lower than the 89.3% of the M2D-LIF model, has only 18.6% of the parameters, achieving an optimal balance between performance and efficiency. Furthermore, compared to the best-performing YOLOv12n model among single-modal RGB methods, our method improves upon mAP50 and mAP50-90 by 0.3% and 4.9% respectively, with only a 170.52% increase in parameters. Compared to the multimodal model YOLOv11-RGBT, our method improves upon mAP50 and mAP50-90 by 0.6% and 5.4% respectively, with only a 79.16% increase in parameters. This fully demonstrates that our method's intelligent fusion strategy at the P4 layer effectively utilizes multimodal complementary information while maintaining the model's lightweight characteristics, providing an ideal solution for real-time pipeline defect detection in edge computing scenarios.

[0088] Example 3: This invention also provides a deep learning-based cross-modal pipeline defect detection system for implementing the method of Embodiment 1, comprising: The image acquisition module is used to acquire the RGB image and depth image of the pipeline to be detected, and to use a dual-branch depth detection network to extract features from the RGB image and depth image respectively to obtain RGB features and depth features. The feature fusion module is used in the high-level semantic layer of the dual-branch deep detection network to perform multimodal feature alignment and adaptive weighted fusion of RGB features and deep features using a feature fusion adapter to obtain fused features. The defect detection module is used to detect defects in pipelines based on fused features and through a detection head.

[0089] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A deep learning-based method for cross-modal pipeline defect detection, characterized in that, include: The RGB and depth images of the pipeline to be detected are obtained, and a dual-branch depth detection network is used to extract features from the RGB and depth images respectively to obtain RGB features and depth features. In the high-level semantic layer of the dual-branch deep detection network, a feature fusion adapter is used to perform multimodal feature alignment and adaptive weighted fusion on the RGB features and the deep features to obtain fused features; Based on the fusion features, the detection head enables the detection of defects in the pipeline.

2. The method according to claim 1, characterized in that, The dual-branch depth detection network adopts a symmetrical parallel encoder-decoder architecture, including an RGB branch and a depth branch. The two branches are aligned and adaptively weighted fused at the P4 level through a feature fusion adapter.

3. The method according to claim 1, characterized in that, The method for obtaining the fusion features includes: Channel alignment and edge enhancement are performed on the depth features to obtain enhanced depth features; Frequency domain separation operations are performed on RGB features and enhanced depth features respectively to extract the low-frequency features of the corresponding RGB mode and the high-frequency features of the depth mode; By utilizing a cross-modal fusion attention mechanism, the low-frequency features of the RGB modality and the high-frequency features of the deep modality are fused across modalities to obtain cross-modal fusion features; The cross-modal fusion features are enhanced using a spatial attention mechanism to obtain multi-scale enhanced features; The quality of RGB and depth features is evaluated by adaptive fusion gating, and three fusion weights for RGB branch, depth branch and cross-modal fusion branch are generated based on the quality evaluation results. Based on the three-way fusion weights, the RGB features, depth features, and multi-scale enhancement features are weighted and summed to obtain the fused features.

4. The method according to claim 3, characterized in that, The method for frequency domain separation of RGB features and enhanced depth features includes: using a 3×3 depthwise separable convolution to perform mean fuzzing on the RGB features and enhanced depth features to obtain low-frequency features of the RGB mode and deep low-frequency features; obtaining high-frequency features of the depth mode based on the residual between the depth features and the deep low-frequency features; wherein, the low-frequency features of the RGB mode correspond to the overall structural information of the pipeline, and the high-frequency features of the depth mode correspond to the geometric edge information of pipeline defects; low-frequency weights and high-frequency weights are generated by introducing a frequency self-gating mechanism to modulate the low-frequency features of the RGB mode and the high-frequency features of the depth mode.

5. The method according to claim 3, characterized in that, Methods for obtaining cross-modal fusion features include: Lightweight channel dimensionality reduction and feature projection are performed on the low-frequency features of the RGB mode and the high-frequency features of the depth mode to obtain the low-frequency feature sequence of the projected RGB mode and the high-frequency feature sequence of the projected depth mode. By utilizing a bidirectional cross-attention architecture based on a multi-head attention mechanism, bidirectional cross-modal attention is calculated using the low-frequency feature sequence of the projected RGB modality and the high-frequency feature sequence of the projected depth modality. The low-frequency features of the RGB modality after bidirectional cross-modal attention enhancement are fused with learnable weights and combined with lightweight residual connections to obtain the cross-modal fused features.

6. The method according to claim 3, characterized in that, Methods for obtaining the three-way fusion weights include: A lightweight quality assessment network is used to score the quality of RGB features and depth features respectively, resulting in RGB quality scores and depth quality scores. Based on the RGB quality score, depth quality score, quality difference, and modal complementarity index, a fusion decision score is generated through a fusion decision network. Based on the RGB quality score, depth quality score, and fusion decision score, and combined with preset basic weights, the initial weights of the RGB branch, depth branch, and cross-modal fusion branch are dynamically generated. The initial weights of the depth branch and the initial weights of the cross-modal fusion branch are modulated with the depth confidence map to obtain spatially adaptive three-way fusion weights; wherein, the depth confidence map is generated by analyzing the edge strength and local variance of the depth features.

7. The method according to claim 3, characterized in that, Methods for obtaining multi-scale enhanced features include: Global average pooling and global max pooling are used to extract global and local defect information of cross-modal fusion features, respectively; Based on global and local defect information, a spatial attention map is generated through a 7×7 convolution. A multi-scale boundary enhancement mechanism is introduced for RGB features to obtain enhanced RGB boundary features; The enhanced RGB boundary features are injected into the cross-modal fusion features using a spatial attention map to obtain the final cross-modal fusion features; Multi-scale enhancement of pipeline defect features at three different scales is achieved by using three 3×3 depthwise separable convolutions with different expansion rates. By employing a multi-scale adaptive fusion strategy, we obtain enhanced features at three different scales and the fusion weights of the final cross-modal fusion features, and then perform multi-scale feature fusion to obtain multi-scale enhanced features.

8. A deep learning-based cross-modal pipeline defect detection system, used to implement the method described in any one of claims 1-7, characterized in that, include: The image acquisition module is used to acquire the RGB image and depth image of the pipeline to be detected, and to use a dual-branch depth detection network to extract features from the RGB image and depth image respectively to obtain RGB features and depth features. The feature fusion module is used to perform multimodal feature alignment and adaptive weighted fusion of the RGB features and the depth features in the high-level semantic layer of the dual-branch deep detection network using a feature fusion adapter to obtain fused features. The defect detection module is used to detect defects in the pipeline based on the fused features and through a detection head.

Images