Underwater pipeline fault detection method and device

Abstract

The present application provides a method and device for underwater pipeline fault detection. The underwater pipeline fault detection method provided by the present application first obtains a set of data for each of multiple locations pre-selected from the underwater pipeline to obtain multiple sets of data; then uses the multiple sets of data to train a detection model; finally, the set of data to be detected is input into the trained detection model to obtain a detection result. The underwater pipeline fault detection method and device provided by the present application, by fusing information from multimodal sensors such as cameras, sonars, and lidars, adds learnable feature hints to the embedded representation of each modality, guides the model to better utilize the information of each modality, and adopts a multi-level structure based on residual low-rank adapters to inject multimodal information into the adapters through a cross-attention mechanism. In this way, the pre-trained detection model can better utilize the information from the multimodal sensors and provide more accurate underwater pipeline fault detection.

Description

Technical Field

[0001] The present application relates to the technical field of fault detection, and in particular to a method and device for detecting underwater pipeline faults. Background Art

[0002] Underwater pipelines are crucial infrastructure for energy transportation, widely used in the transport of oil, natural gas, chemicals, and other energy sources. In underwater environments, pipelines often face complex geographical conditions and harsh environmental factors, such as water erosion, high pressure at depth, and corrosion caused by seawater. These factors increase the difficulty and risk of pipeline inspection.

[0003] Traditional underwater pipeline inspection technologies primarily rely on single sensor devices, such as underwater robots, remotely operated vehicles (ROVs), and sonar detectors. However, single sensor devices have limited detection capabilities and cannot fully reflect the health of pipelines, making them prone to missing pipeline faults. Summary of the Invention

[0004] In view of this, the present application provides a method and device for underwater pipeline fault detection, so as to achieve accurate and efficient underwater pipeline detection.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] A first aspect of the present application provides a method for detecting underwater pipeline faults, the method comprising:

[0007] For each of a plurality of positions pre-selected from the underwater pipeline, a set of data of the position is acquired to obtain multiple sets of data; the set of data includes a visual image, an acoustic image and a laser point cloud;

[0008] The detection model is trained using the multiple groups of data; the detection model includes a first feature extraction module, a multimodal fusion module, a second feature extraction module and a fault classifier; the first feature extraction module is used to extract the initial features of each modality, and splice the learnable prompt features corresponding to the modality on the initial features of each modality to form spliced visual features, spliced acoustic features and spliced point cloud features; the multimodal fusion module is used to fuse the spliced features to obtain the final fusion features; the second feature extraction module includes multiple encoding modules, each encoding module includes multiple cascaded encoding layers, a low-rank adapter connected to the output end of the second-to-last encoding layer of the multiple encoding layers, and a weighted fusion layer connected to the output end of the first-to-last encoding layer of the multiple encoding layers and the output end of the low-rank adapter; the first encoding layer of the multiple encoding layers of the first encoding module is used to receive the visual splicing feature, and the input end of the low-rank adapter of the first encoding module is connected to the output end of the multimodal fusion module; the low-rank adapters of two adjacent encoding modules are connected, and the weighted fusion layer of the previous encoding module in the two adjacent encoding modules is connected to the first encoding layer of the next encoding module;

[0009] A set of data to be tested is input into the trained detection model to obtain the detection results.

[0010] A second aspect of the present application provides a pipeline fault detection device, comprising an acquisition module, a training module, and a detection module; wherein the acquisition module is configured to acquire a set of data for each of a plurality of pre-selected positions in an underwater pipeline, thereby obtaining multiple sets of data; a set of data includes a visual image, an acoustic image, and a laser point cloud;

[0011] The training module is used to train the detection model using the multiple groups of data; the detection model includes a first feature extraction module, a multimodal fusion module, a second feature extraction module and a fault classifier; the first feature extraction module is used to extract the initial features of each modality, and splice the learnable prompt features corresponding to the modality on the initial features of each modality to form spliced visual features, spliced acoustic features and spliced point cloud features; the multimodal fusion module is used to fuse the spliced features to obtain the final fusion features; the second feature extraction module includes multiple encoding modules, each encoding module includes multiple cascaded encoding layers, a low-rank adapter connected to the output end of the second-to-last encoding layer of the multiple encoding layers, and a weighted fusion layer connected to the output end of the first-to-last encoding layer of the multiple encoding layers and the output end of the low-rank adapter; the first encoding layer of the multiple encoding layers of the first encoding module is used to receive the visual splicing feature, and the input end of the low-rank adapter of the first encoding module is connected to the output end of the multimodal fusion module; the low-rank adapters of two adjacent encoding modules are connected, and the weighted fusion layer of the previous encoding module in the two adjacent encoding modules is connected to the first encoding layer of the next encoding module;

[0012] The detection module is used to input a set of to-be-detected data into the trained detection model to obtain a detection result.

[0013] The underwater pipeline fault detection method and device provided in this application have a first feature extraction module that extracts features from visual images, acoustic images, and laser point clouds respectively, and splices learnable prompt features based on the initial features, so that data of different modalities have better expression capabilities before fusion. The multimodal fusion module adopts a learnable fusion method to make the information of visual, acoustic, and point cloud data complementary, effectively enhancing the extraction capability of key fault features and overcoming the limitations of a single modality that may be affected by the environment. Furthermore, the second feature adopts a hierarchical coding structure based on a low-rank adapter, so that information is gradually optimized in coding layers of different depths, making the semantics of the features more complete and enhancing the feature expression capability. In this way, the three work together, so that the detection model can still accurately identify different types of pipeline faults in a variety of underwater environments (insufficient lighting, noise interference, etc.), thereby improving detection accuracy.

[0014] In addition, the parameters of the encoding layer and weighting layer are fixed to ensure the stability of feature extraction, while the parameters of the low-rank adapter and multimodal fusion module are learnable, so that the model can be adaptively optimized in different environments, improve generalization ability, and improve detection accuracy and robustness while reducing training parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of Example 1 of the underwater pipeline fault detection method provided by this application;

[0016] Figure 2 A schematic diagram of the structure of a detection model provided by an exemplary embodiment of the present application;

[0017] Figure 3 This is a schematic structural diagram of a first feature extraction module according to an exemplary embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of the internal structure of a low-rank adapter shown in an exemplary embodiment of the present application;

[0019] Figure 5 This is a schematic diagram of Example 1 of the underwater pipeline fault detection device provided in this application. DETAILED DESCRIPTION

[0020] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0021] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0023] Specific embodiments are given below to introduce the technical solutions of the present application in detail.

[0024] Figure 1 This is a flow chart of Example 1 of the underwater pipeline fault detection method provided by this application. Figure 2 This is a structural diagram of a detection model shown in an exemplary embodiment of this application. Please also refer to Figure 1 and Figure 2 The method provided in this embodiment may include:

[0025] S101. For each of a plurality of positions pre-selected from an underwater pipeline, a set of data for the position is acquired to obtain multiple sets of data; one set of data includes a visual image, an acoustic image, and a laser point cloud.

[0026] Specifically, the underwater pipeline fault detection method and device provided in the present application can perform fault detection on underwater pipelines that transport natural energy. For example, it can perform transportation safety detection on underwater transportation pipelines that transport natural energy such as oil or natural gas.

[0027] It should be noted that underwater pipelines for transporting natural energy usually extend for several kilometers. Underwater pipelines at different locations may have significantly different types and severity of faults in various parts of the underwater pipelines due to factors such as environmental pressure, degree of corrosion, mechanical stress or geological activity. Therefore, during the inspection process, multiple locations of the underwater pipeline can be selected and inspected separately. In this way, it can ensure that high-risk areas in the underwater pipeline are effectively covered, and avoid missing overall hidden dangers inside the underwater pipeline due to local inspections.

[0028] It should be noted that the multiple locations pre-selected from the underwater pipeline are selected based on actual needs and are not limited in this application. For example, in one embodiment, when an underwater pipeline transports natural energy such as oil or natural gas, leakage is likely to occur at the weld joints of the underwater pipeline. Therefore, weld joints throughout the underwater pipeline can be selected as multiple locations for fault detection.

[0029] Furthermore, after selecting multiple locations where fault detection of underwater pipelines is required, for each of the multiple locations, a visual image, an acoustic image, and a laser point cloud at the location are obtained as a set of data, thereby obtaining multiple sets of data corresponding to the multiple locations.

[0030] It should be noted that, in specific implementations, appropriate data acquisition tools can be selected based on actual needs to collect visual images, acoustic images, and laser point clouds at each location, and this application does not limit these tools. For example, in one embodiment, a pressure-resistant underwater optical camera is used to collect visual images of the underwater pipeline, an ultrasonic sensor is used to collect acoustic images of the underwater pipeline, and a lidar sensor is used to collect laser point clouds of the underwater pipeline.

[0031] Optionally, in a possible implementation, after collecting multiple sets of data, each set of data in the multiple sets of data may be preprocessed to remove noise therein.

[0032] It should be noted that the visual images, acoustic images and laser point clouds in each set of data are all collected using sensor equipment. The data collected by the sensors underwater are often susceptible to various environmental interferences, resulting in a large amount of noise in the collected raw data. Therefore, the collected raw data can be preprocessed to improve the quality of the visual images, acoustic images and laser point clouds to ensure the accuracy of subsequent detection.

[0033] Optionally, in a possible implementation, each set of data may be preprocessed using the following method:

[0034] For the visual image, determining a denoising algorithm and parameters of the denoising algorithm according to features of the visual image, and performing denoising processing on the visual image based on the determined denoising algorithm and parameters of the denoising algorithm;

[0035] For acoustic images, common features are determined based on the features of the visual image and the features of the acoustic image, and the acoustic image is denoised based on the common features;

[0036] For point cloud data, isolated and discrete points are removed based on semantic similarity and semantic coherence.

[0037] Specifically, when performing denoising on a visual image, the denoising algorithm and its parameters can be determined based on characteristics such as the local contrast of the visual image (the local contrast of the visual image is represented by LC) and the noise standard deviation (the noise standard deviation of the visual image is represented by σ).

[0038] For example, in one possible implementation, when image features are low-contrast (local contrast LC < 0.3 for low-contrast image features) and high-noise (noise standard σ > 15 for high-noise image features), a non-local means denoising algorithm is used to denoise the visual image. The parameters used may be: a search window of 21×21, a similarity window of 7×7, and an attenuation coefficient h = 0.6σ. Furthermore, when image features are high-contrast and low-noise, an adaptive median filtering denoising algorithm is used to denoise the visual features. The parameters used may be a window expansion of 7×7 and a threshold of 255.

[0039] Furthermore, when performing denoising on the acoustic image, common features may be determined based on the features of the visual image and the features of the acoustic image to perform denoising on the acoustic image.

[0040] It should be noted that the common features of visual and acoustic images refer to the characteristic expressions in both visual and acoustic images that represent the true physical properties of underwater pipelines. For example, common features in visual and acoustic images include structural features, anomaly features, and geometric features. Structural features include the pipeline axis, girth welds, and the location of supporting structures. Anomaly features include corroded areas on underwater pipelines, which appear as color spots visually and as enhanced scattering acoustically. Geometric features include three-dimensional morphological features such as pipe diameter changes and concavities. The acoustic images are then denoised based on the obtained common features.

[0041] For example, in one possible implementation, the pipeline axis can be extracted from the visual image through edge detection and Hough transform. In the acoustic image, the reflection trajectory of the pipeline can be extracted through reflection pattern and echo analysis. This trajectory represents the pipeline axis. The noise type (including Gaussian noise, scattered noise, speckle noise, etc.) is then determined based on the characteristics of the acoustic image. The pipeline axis parameters in the visual image (including direction and position) are then aligned with those in the acoustic image to ensure spatial consistency. Furthermore, the acoustic image is denoised based on the pipeline axis parameters. Specifically, for example, a mask region containing the pipeline structure (i.e., a strong reflection region near the axis) is constructed with the pipeline axis extracted from the acoustic image as the center to protect the pipeline region. Then, a noise denoising method using non-local mean filtering or guided filtering is used to selectively denoise the noise region outside the mask region.

[0042] Furthermore, when denoising point cloud data, semantic segmentation can be used to classify the point cloud data and mark the underwater pipeline body and other objects (such as water bodies, attachments, etc.); then, the neighborhood information of each point cloud in the point cloud data is used to calculate the semantic consistency of each point cloud. If the semantic similarity between a point cloud and other point clouds in its neighborhood is lower than a preset threshold, it will be marked as an isolated point; then, a clustering algorithm is used to cluster points of the same category to remove small clusters and scattered discrete points.

[0043] It should be noted that in this embodiment, different methods are used to denoise visual images, acoustic images, and point cloud data. The denoising algorithm and parameters of visual images are adaptively determined based on their characteristics, which can more accurately remove noise while retaining effective information. Acoustic image denoising is based on common features with visual images, making cross-modal denoising more robust. Point cloud data denoising takes semantic information into account to avoid simple statistical filtering methods that mistakenly delete important points, thereby improving the quality of point cloud data. The denoising methods for visual, acoustic, and point cloud data are adapted to their characteristics, which can enhance adaptability in different environments. In addition, acoustic image denoising not only relies on its own characteristics, but also combines visual image features to determine common features, making denoising more stable and physically consistent.

[0044] In summary, this solution adopts a highly targeted denoising strategy for visual, acoustic, and point cloud data, which improves the denoising effect, data consistency, computational efficiency, and system robustness, providing high-quality data support for subsequent processing.

[0045] S102: Utilize the multiple groups of data to train a detection model.

[0046] For details, see Figure 2,The detection model includes a first feature extraction module, a multimodal fusion module, a second feature extraction module and a fault classifier.

[0047] Among them, the first feature extraction module is used to extract features from the visual image, acoustic image and laser point cloud respectively to obtain initial visual features, initial acoustic features and initial point cloud features, and splice the learnable prompt features corresponding to the modality on the initial features of each modality (the learnable prompt features are recorded as tokens) to form spliced visual features, spliced acoustic features and spliced point cloud features.

[0048] Optional, Figure 3 This is a schematic diagram of the structure of the first feature extraction module shown in an exemplary embodiment of the present application. Figure 3 In a possible implementation, the first feature extraction module includes a visual image encoder, an acoustic image encoder, and a laser point cloud encoder; the visual image encoder is used to extract features from the visual image; the acoustic image encoder is used to extract features from the acoustic image; and the laser point cloud encoder is used to extract features from the laser point cloud;

[0049] The network parameters of the visual image encoder and the laser point cloud encoder are fixed, and the network parameters of the acoustic image encoder are learnable.

[0050] Specifically, the detection model uses visual images as the backbone, and the visual large model (CLIP model, based on the Transformer architecture) that processes visual image information is used as the backbone network for fault detection. The visual image encoder uses the convolution part of the backbone network in the visual large model to extract features from the visual image information through the visual image encoder to obtain the initial visual features. .

[0051] Furthermore, in a possible implementation, the acoustic image encoder uses a multi-layer residual-based convolutional network to extract features from the acoustic image and obtain the initial acoustic features. In addition, the laser point cloud encoder uses the VoxelNet network to extract the 3D voxel features in the laser point cloud data. After that, the obtained 3D voxel features are flattened along the Z axis to generate a bird's-eye view, and the initial point cloud features can be obtained. .

[0052] It should be noted that, Indicates the number of input samples for each training. represents the length of the feature map, represents the width of the feature map, The number of channels representing features. For example, the initial visual features , then the number of visual image samples representing a single input is 8, the length of the output initial visual feature map is 20, the width of the output initial visual feature map is 15, and the number of channels of the output initial visual feature is 2048.

[0053] In this embodiment, the network parameters of the visual image encoder and the laser point cloud encoder are fixed, and the network parameters of the acoustic image encoder are learnable.

[0054] Specifically, the network parameters refer to the weights and biases of the encoder. The fixed network parameters of the visual image encoder and the laser point cloud encoder mean that the weights and biases inside the visual image encoder and the laser point cloud encoder will not be updated during the training process of the detection model, retaining their existing feature extraction capabilities; the learnable network parameters of the acoustic image encoder mean that their weights and biases will be continuously updated during the training process to adapt to the needs of the current task and optimize their own feature extraction capabilities.

[0055] For further information, please refer to Figure 4 After the first feature extraction module extracts the visual image, acoustic image and point cloud data respectively to obtain the initial visual features, initial acoustic features and initial point cloud features, the dimensions of the initial visual features, initial acoustic features and initial point cloud features are flattened into a one-dimensional length dimension to obtain the embedding tokens corresponding to each modality (each modality includes vision, acoustics and point cloud). After that, the prompt tokens corresponding to each modality are spliced on the embedded tokens of each modality to form the spliced visual features, spliced acoustic features and spliced point cloud features.

[0056] It's important to note that the hint tokens corresponding to each modality instruct the detection model to identify the specific modality when processing the input. Subsequently, the embedding tokens for each modality are concatenated with the hint token to create a new token that incorporates both the modality hint and the modality embedding. This allows the detection model to effectively integrate information from different modalities, helping it understand the relationships between them and enabling more effective multimodal reasoning and processing.

[0057] Referring to the previous description, it can be understood that the spliced visual features, spliced acoustic features and spliced point cloud features represent the new tokens corresponding to the image modality, acoustic modality and point cloud modality respectively; among them, the spliced visual features are recorded as , the spliced acoustic features are recorded as , the spliced point cloud features are recorded as , It represents the length of the new token, that is, the dimension of the concatenated features.

[0058] It should be noted that the modal hint tokens corresponding to each modality are learnable parameters. Through continuous training of the modal hint tokens, the token values can be adjusted during training to better help the model understand and integrate information from different modalities, thereby improving the detection model's multimodal processing capabilities. For example, the hint tokens for the visual modality are gradually adjusted during training to help better integrate visual features with features from other modalities during splicing; the hint tokens for the acoustic modality are gradually adjusted during training to help the model better recognize and understand audio data; and the hint tokens for the point cloud modality are continuously adjusted during training to learn how to identify the best point cloud data.

[0059] For further information, please refer to Figure 2 The detection model also includes a multimodal fusion module, which is used to fuse the spliced features to obtain the final fusion features, that is, to fuse the spliced visual features, the spliced acoustic features and the spliced point cloud features to obtain the final fusion features.

[0060] Specifically, the multimodal fusion module is specifically used to fuse the spliced visual features, the spliced acoustic features and the spliced point cloud features to obtain initial fusion features, perform nonlinear transformation on the initial fusion features to obtain enhanced fusion features, and perform weighted processing on the spliced visual features and the enhanced fusion features to obtain final fusion features.

[0061] It can be understood that the multimodal fusion module may include a fusion layer, a feedforward network and a fully connected layer. The fusion layer is specifically used to fuse the spliced visual features, the spliced acoustic features and the spliced point cloud features to obtain initial fusion features; the feedforward network is used to perform nonlinear transformation on the initial fusion features to obtain enhanced fusion features; the fully connected layer is used to perform weighted processing on the spliced visual features and the enhanced fusion features to obtain final fusion features.

[0062] The following describes the specific implementation principles within the multimodal fusion module:

[0063] Specifically, for example, the spliced features can be directly fused based on a weighted or spliced manner to obtain an initial fused feature. In addition, in one possible implementation, when fusing the spliced features, the spliced visual features and the spliced acoustic features can be fused based on a first cross-attention module, using the spliced visual features as a query and the spliced acoustic features as a key and a value, to obtain a first fused feature. Furthermore, based on a second cross-attention module, the first fused feature and the spliced point cloud features can be fused based on the first fused feature as a query and the spliced point cloud features as a key and a value, to obtain the initial fused feature.

[0064] Optionally, in a possible implementation, fusing the spliced visual features, the spliced acoustic features, and the spliced point cloud features to obtain an initial fused feature includes:

[0065] Step 1: For each modal data, calculate the confidence level of the modal data based on the feature information of the modal data.

[0066] In this step, for visual image data, its confidence can be calculated based on characteristic information such as image clarity, contrast, and noise level. For example, the Laplace variance can be used to calculate the degree of image blur as the confidence. For another example, a CNN model can be used to learn and extract image quality features from visual image features and predict their confidence.

[0067] For acoustic image data, the quality of the acoustic image can be evaluated based on the signal-to-noise ratio or spectral energy distribution of the acoustic image data, and its confidence level can be calculated. For example, after calculating the short-time Fourier transform, the proportion of high-energy areas can be calculated to determine the signal quality, and the confidence level can be calculated based on the signal quality.

[0068] For point cloud image data, the confidence of the point cloud modal data can be calculated based on the sparsity, point density, reconstruction completeness, etc. For example, the density histogram of the point cloud can be calculated. If the density is low, it means that the point cloud data is relatively sparse and the confidence is low.

[0069] Step 2: Normalize the confidence of various modal data and determine the normalization result as the weighted weight corresponding to the modal data.

[0070] In this step, the confidence levels of the image modal data, acoustic modal data, and point cloud modal data are normalized respectively.

[0071] It should be noted that the Softmax method can be used to normalize the confidence of the three modal data. Assume that the confidence of image modal data, acoustic modal data and point cloud modal data are 、 、 , then the following formula is normalized:

[0072] ;

[0073] ;

[0074] ;

[0075] in, is the weighted weight corresponding to the image modality data; is the weighted weight corresponding to the acoustic modal data; is the weighted weight corresponding to the point cloud modal data.

[0076] Step 3: Fusing the spliced visual features, the spliced acoustic features, and the spliced point cloud features according to the weighted weights corresponding to each modal data to obtain an initial fused feature.

[0077] In this step, the initial fusion features are obtained by fusion of the spliced visual features, spliced acoustic features, and spliced point cloud features using the weighted summation method according to the following formula:

[0078] ;

[0079] in, is the initial fusion feature; To stitch visual features; To splice acoustic features; It is the feature of splicing point cloud.

[0080] It should be noted that in this embodiment, when the initial fusion features are obtained based on the spliced visual features, spliced acoustic features and spliced power features, an adaptive weighting mechanism is introduced to dynamically adjust the weighting weights according to the confidence of different modal data. This can improve the fusion effect of multimodal data, improve the accuracy of the initial fusion features, and make the detection model more adaptable to changes in the quality of different modal data, thereby improving the robustness and accuracy of underwater pipeline fault detection.

[0081] Furthermore, in a possible implementation, performing a nonlinear transformation on the initial fusion feature to obtain an enhanced fusion feature includes:

[0082] Based on the feedforward network, the initial fusion features are nonlinearly transformed using the first formula to obtain enhanced fusion features; the first formula is:

[0083] ;

[0084] Among them, the is the initial fusion feature;

[0085] described for the enhanced fusion feature;

[0086] described and stated is the learnable parameter matrix, and stated is a learnable bias term, is a non-linear activation function.

[0087] Furthermore, in a possible implementation, the spliced visual features and the enhanced fusion features are weighted to obtain a final fusion feature, including:

[0088] Based on the fully connected layer, the spliced visual features and the enhanced fusion features are weighted using the second formula to obtain the final fusion feature; the second formula is:

[0089] ;

[0090] Among them, the is the final fusion feature;

[0091] described is the splicing visual feature;

[0092] described for the enhanced fusion feature;

[0093] described is the learnable parameter vector,

[0094] It should be noted that visual, acoustic, and point cloud data each provide information of different dimensions. After fusing the three, the complementarity of different modal features can be achieved. Furthermore, through nonlinear transformations, deeper feature relationships can be learned, the expressive power of features can be enhanced, and the fused features can be made more abstract and rich. Finally, by further fusing visual features and features after nonlinear changes, the visual features cannot be completely replaced by features of other modalities during the final fusion. The original visual information can be maintained, so that visual information still occupies a dominant position, ensuring that visual perception is not weakened. Additional information such as depth and material can be supplemented through features of other modalities to improve overall perception. In addition, through learnable parameters, the network is allowed to automatically adjust the weights of features, which not only improves the ability to interact with information, but also makes the multimodal contribution different in different scenarios, thereby improving generalization ability.

[0095] For further information, please refer to Figure 2The detection model also includes a second feature extraction module, which includes multiple encoding modules, each encoding module including multiple cascaded encoding layers, a low-rank adapter connected to the output end of the second-to-last encoding layer of the multiple encoding layers, and a weighted fusion layer connected to the output end of the first-to-last encoding layer of the multiple encoding layers and the output end of the low-rank adapter; the first encoding layer of the multiple encoding layers of the first encoding module is used to receive visual splicing features, and the input end of the low-rank adapter of the first encoding module is connected to the output end of the multimodal fusion module; the low-rank adapters of two adjacent encoding modules are connected, and the weighted fusion layer of the previous encoding module in the two adjacent encoding modules is connected to the first encoding layer of the next encoding module; the parameters of the encoding layer and the weighted layer in each encoding module are fixed, and the parameters of the low-rank adapter are learnable.

[0096] It should be noted that the specific number of encoding modules is set according to actual needs and is not limited in this embodiment. In addition, the encoding layer in the encoding module is a Transformer encoding layer. That is, the model part involved in the logical line from visual image to fault classifier can adopt an existing pre-trained large visual model. Furthermore, the multiple Transformer encoding layers in this model part are divided into multiple parts, each of which constitutes an encoding module. Furthermore, for this encoding module, the last layer of Transformer encoding layer is taken for adaptation processing, forming a multi-level adaptation processing, that is, the output of the second-to-last Transformer encoding layer is simultaneously input to the low-rank adapter.

[0097] Specifically, Figure 4 This is a schematic diagram of the internal structure of a low-rank adapter shown in an exemplary embodiment of the present application. Please refer to Figure 4 ,In a possible implementation, the low-rank adapter includes a cross-attention module, a dimensionality reduction module, a nonlinear processing module, and a dimensionality increase module;

[0098] The cross attention module is configured to use the main line feature from the encoding layer as a query and the secondary line feature from the multimodal fusion module or the previous low-rank adapter as a key and a value, and fuse the main line feature and the secondary line feature to obtain a fused feature;

[0099] The dimension reduction module is used to perform dimension reduction processing on the fusion feature to obtain a dimension-reduced fusion feature;

[0100] The nonlinear processing module is used to perform nonlinear processing on the dimension reduction fusion features to obtain processed features;

[0101] The dimension-increasing module is used to process the processed features to obtain dimension-increasing fusion features; wherein the dimension of the dimension-increasing fusion features is the same as the dimension of the fusion features.

[0102] Specifically, the crisscross attention module in the low-rank adapter receives Figure 2 The main-line features of the transformer encoder in the multimodal fusion module (for the convenience of explanation, the features output by the transformer encoding layer are recorded as the main-line features, and the main-line features are the features output by the penultimate transformer encoding layer in each encoding module) and the secondary-line features from the low-rank adapter in the multimodal fusion module or the pre-order encoding module (for the convenience of explanation, the features output by the low-rank adapter in the multimodal fusion module or the pre-order encoding module are recorded as secondary-line features) are fused through the attention mechanism (where the main-line features are used as the query and the secondary-line features are used as the key and value) to obtain the fused features.

[0103] As described above, primary features are features directly transmitted from the backbone network in the detection model, while secondary features come from low-rank adapters. These are auxiliary information in the detection model, providing additional feature information for the primary features through cross-modal or cross-layer approaches. In this embodiment, the features from the transformer encoding layer are primary features, while the features from the low-rank adapters in the multimodal fusion module or pre-order encoding module are secondary features.

[0104] Furthermore, the dimensionality reduction module in the low-rank adapter reduces the dimensionality of the fused features to reduce their complexity. For example, a single fully connected layer can be used as a dimensionality reduction module to reduce the feature dimension C of the fused features to Csmall. The nonlinear processing module in the low-rank adapter then performs nonlinear processing on the reduced features to obtain processed features, which have stronger nonlinear expression capabilities.

[0105] It should be noted that the nonlinear processing module in the low-rank adapter can be selected according to actual needs and is not limited in this application. For example, in one embodiment, a ReLU activation function can be used for nonlinear processing.

[0106] Furthermore, dimensionality increase processing is performed through the dimensionality increase module in the low-rank adapter. For example, the feature dimension of the processed module can be restored from Csmall to dimension C using a fully connected layer to obtain the dimensionality increase fusion feature. In this way, it can be ensured that the output dimensionality increase fusion feature is compatible with the backbone network during data transmission.

[0107] Finally, see Figure 2For a coding module, the upgraded features output by the low-rank adapter are fused with the output of the last transformer coding layer in the coding module through a weighted fusion layer. After fusion, they are input into the first coding layer of the next coding module. At the same time, the upgraded features of the low-rank adapter are also input into the low-rank adapter within the next coding module for further processing. In other words, for a low-rank adapter, its input includes the main line features input from the coding layer and the amplitude line features input from the multimodal fusion module or the previous low-rank adapter.

[0108] Referring to the previous description, in this embodiment, by setting the second feature extraction module to a multi-level structure based on the residual low-rank adapter, multimodal information is injected into the low-rank adapter through the cross-attention mechanism, so that the pre-trained large visual model can better utilize multimodal information, effectively adapt to the underwater pipeline detection task, improve the detection accuracy and robustness, and reduce the training parameters.

[0109] It should be noted that the parameters of the encoding layer and weighted layer in each encoding module are fixed, which means that the weights of the encoding layer and weighted layer are not updated during the training phase of the detection model. These parameters are pre-trained before the detection model is trained to have general representation capabilities in a variety of scenarios. Even without weight updates, stable feature extraction capabilities can be achieved; and the parameters of the low-rank adapter are learnable, which means that in the encoding module, the parameters of the low-rank adapter will be gradually updated during the training process of the detection model, so that adaptive adjustment to specific tasks can be achieved by adjusting the parameters of the low-rank adapter while maintaining the stability of the visual backbone network.

[0110] Optionally, in one possible implementation, variable-capacity low-rank adapters are used in different encoding modules (i.e., the ranks of low-rank adapters in different transform encoding modules are different) to adapt to information requirements at different levels. This can effectively improve the flexibility of feature transformation.

[0111] Furthermore, in a possible implementation, the rank of the low-rank adapter of each coding module is determined based on features input into the low-rank adapter.

[0112] Specifically, the rank of the low-rank adapter is the dimension obtained by reducing the fusion features by the dimensionality reduction module inside the low-rank adapter. It should be noted that the rank of the low-rank adapter will affect the efficiency and performance of the low-rank adapter. In this embodiment, selecting low-rank adapters of different ranks for different features can maximize the balance between the efficiency and performance of the low-rank adapter and adapt to various scenarios.

[0113] Optionally, in a possible implementation, a method for determining the rank of a low-rank adapter of each coding module includes:

[0114] Step 1: Use a multi-layer perceptron to process the features input to the low-rank adapter to obtain the initial prediction value.

[0115] In this step, the features of the low-rank adapter are input into a multi-layer perceptron, which extracts implicit complexity information from the input features (complexity information includes feature entropy, activation sparsity, cross-modal conflict strength, etc.), and then maps the obtained complexity information into an initial predicted value of the rank.

[0116] Step 2: Activate the initial prediction value using an activation function to obtain the rank of the low-rank adapter.

[0117] In this step, the initial predicted value of the rank obtained in step 1 is input into the selected activation function (the selected activation function can be the Sigmoid function), and the activation function activates the initial predicted value and adjusts the range of the initial predicted value to ensure that the rank value is within a reasonable range, and finally obtains the rank of the low-rank adapter.

[0118] It should be noted that, in this embodiment, each encoding module receives feature information at different levels, and the expressive power and information density of these features vary between levels. In order to adapt to different information requirements, the rank of the low-rank adapter is dynamically determined based on the input features of each encoding module. Through this dynamic adjustment mechanism, a larger rank can be used in a higher-level encoding module to effectively capture global features, while a smaller rank can be used in a lower-level encoding module to avoid excessive computational overhead. In this way, not only can the expressive power of the features be improved, but also the computational overhead can be taken into account at the same time.

[0119] S103: Input a set of data to be tested into the trained detection model to obtain a detection result.

[0120] Specifically, after the previous training of the detection model, in this step, a set of data to be detected can be input into the trained detection model after denoising, and the detection model outputs the result.

[0121] Specifically, the first feature extraction module in the detection model first extracts the initial visual features of the visual image, the initial acoustic features of the acoustic image and the initial point cloud features of the point cloud data in the data to be detected; then, the learned prompt features (token) corresponding to each modality are spliced on the initial features of the modality to form spliced visual features, spliced acoustic features and spliced point cloud features; then, the multimodal fusion module first performs weighted fusion on the spliced visual features, spliced acoustic features and spliced point cloud features to obtain the initial fusion features, then performs nonlinear slowing on the initial fusion features to obtain enhanced fusion features, and finally, performs weighted processing on the spliced visual features and enhanced fusion features to obtain the final fusion features; then, The final fusion feature is input into the low-rank adapter in the first encoding module, and the spliced visual feature is input into the first transform coding layer in the encoding module. It is processed layer by layer through the internal transform coding layer. After that, the output of the penultimate transform coding layer in the encoding module is input into the low-rank adapter. The low-rank adapter outputs the up-dimensional fusion feature based on the features transmitted from the multimodal fusion modality and the features transmitted from the penultimate transform coding layer. The up-dimensional fusion feature and the features output by the last transform coding layer are input into the weighted fusion layer, which performs weighted fusion to obtain the result.

[0122] Furthermore, the result is used as an input feature and input again into the first transform coding layer in the next coding module. At the same time, the dimensionality-increased fusion feature output by the low-rank adapter in the previous coding module is also input into the low-rank adapter in the next coding module; ...; After cyclic processing of multiple coding modules, the output of the weighted fusion layer of the last coding module is input into the fault classifier, and the fault classifier outputs the final detection result.

[0123] It should be noted that the fault classifier determines the fault type according to the input content by setting up an experience database; wherein, the experience database can use an existing database or be set by the operator based on experience, and is not limited in this application.

[0124] In this embodiment, by fusing information from multimodal sensors such as cameras, sonar, and lidar, and adding learnable feature cues to the embedded representation of each modality, the model can be guided to better utilize the information of each modality; and a multi-level structure based on a residual low-rank adapter is adopted to inject multimodal information into the low-rank adapter through a cross-attention mechanism. In this way, the pre-trained detection model can better utilize the information from multimodal sensors and effectively adapt to underwater pipeline inspection tasks. This method reduces training parameters while improving detection accuracy and robustness.

[0125] In the underwater pipeline fault detection method provided in this embodiment, the first feature extraction module extracts features from visual images, acoustic images, and laser point clouds respectively, and splices learnable prompt features based on the initial features, so that the data of different modalities have better expressive ability before fusion. The multimodal fusion module adopts a learnable fusion method to make the information of visual, acoustic, and point cloud data complementary, effectively enhancing the ability to extract key fault features and overcoming the limitations of a single modality that may be affected by the environment. Furthermore, the second feature adopts a hierarchical coding structure based on low-rank adapters, allowing information to be gradually optimized in coding layers of different depths, making the feature semantics more complete and enhancing the feature expressiveness. In this way, the three work together, allowing the detection model to still accurately identify different types of pipeline faults in various underwater environments (such as insufficient lighting and noise interference), thereby improving detection accuracy.

[0126] In addition, the parameters of the encoding layer and weighting layer are fixed to ensure the stability of feature extraction, while the parameters of the low-rank adapter and multimodal fusion module are learnable, so that the model can be adaptively optimized in different environments, improve generalization ability, and improve detection accuracy and robustness while reducing training parameters.

[0127] In summary, the method provided in this embodiment, through multimodal data fusion and a hierarchical coding structure, solves the challenges of multimodal fusion and environmental interference faced by traditional underwater pipeline fault detection, and achieves efficient, accurate, and robust underwater pipeline fault detection.

[0128] Corresponding to the aforementioned embodiment of an underwater pipeline fault detection method, the present application also provides an embodiment of an underwater pipeline fault detection device.

[0129] Figure 5 This is a schematic diagram of the first embodiment of the underwater pipeline fault detection device provided by this application. Figure 5 The device provided in this embodiment includes an acquisition module 510, a training module 520, and a detection module 530. The acquisition module 510 is configured to acquire a set of data for each of a plurality of positions pre-selected from the underwater pipeline, thereby obtaining multiple sets of data. The set of data includes a visual image, an acoustic image, and a laser point cloud.

[0130] The training module 520 is used to train the detection model using the multiple groups of data; the detection model includes a first feature extraction module, a multimodal fusion module, a second feature extraction module and a fault classifier; the first feature extraction module is used to extract the initial features of each modality, and splice the learnable prompt features corresponding to the modality on the initial features of each modality to form spliced visual features, spliced acoustic features and spliced point cloud features; the multimodal fusion module is used to fuse the spliced features to obtain the final fusion features; the second feature extraction module includes multiple encoding modules, each encoding module includes multiple cascaded encoding layers, a low-rank adapter connected to the output end of the second-to-last encoding layer of the multiple encoding layers, and a weighted fusion layer connected to the output end of the first-to-last encoding layer of the multiple encoding layers and the output end of the low-rank adapter; the first encoding layer of the multiple encoding layers of the first encoding module is used to receive the visual splicing feature, and the input end of the low-rank adapter of the first encoding module is connected to the output end of the multimodal fusion module; the low-rank adapters of two adjacent encoding modules are connected, and the weighted fusion layer of the previous encoding module in the two adjacent encoding modules is connected to the first encoding layer of the next encoding module;

[0131] The detection module 530 is used to input a set of to-be-detected data into the trained detection model to obtain a detection result.

[0132] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0133] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0134] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0135] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for detecting underwater pipeline faults, characterized in that: The method comprises: For each of a plurality of positions pre-selected from the underwater pipeline, a set of data of the position is acquired to obtain multiple sets of data; the set of data includes a visual image, an acoustic image and a laser point cloud; The detection model is trained using the multiple groups of data; the detection model includes a first feature extraction module, a multimodal fusion module, a second feature extraction module and a fault classifier; the first feature extraction module is used to extract the initial features of each modality, and splice the learnable prompt features corresponding to the modality on the initial features of each modality to form spliced visual features, spliced acoustic features and spliced point cloud features; the multimodal fusion module is used to fuse the spliced features to obtain the final fusion features; the second feature extraction module includes a plurality of encoding modules, each encoding module includes a plurality of cascaded encoding layers, a low-rank adapter connected to the output end of the second-to-last encoding layer of the plurality of encoding layers, and an output end of the first-to-last encoding layer of the plurality of encoding layers and the output end of the low-rank adapter The weighted fusion layers are connected simultaneously; the first coding layer of the multiple coding layers of the first coding module is used to receive the visual splicing features, and the input end of the low-rank adapter of the first coding module is connected to the output end of the multimodal fusion module; the low-rank adapters of two adjacent coding modules are connected, and the weighted fusion layer of the previous coding module in the two adjacent coding modules is connected to the first coding layer of the next coding module; the multimodal fusion module is specifically used to fuse the spliced visual features, the spliced acoustic features and the spliced point cloud features to obtain initial fusion features, and perform nonlinear transformation on the initial fusion features to obtain enhanced fusion features, and perform weighted processing on the spliced visual features and the enhanced fusion features to obtain final fusion features; The performing of nonlinear transformation on the initial fusion features to obtain enhanced fusion features includes: Based on the feedforward network, the initial fusion features are nonlinearly transformed using the first formula to obtain enhanced fusion features; the first formula is: ； Among them, the is the initial fusion feature; described for the enhanced fusion feature; described and stated is the learnable parameter matrix, and stated is a learnable bias term, is a nonlinear activation function; A set of data to be tested is input into the trained detection model to obtain the detection results.

2. The method according to claim 1, characterized in that The first feature extraction module includes a visual image encoder, an acoustic image encoder and a laser point cloud encoder; the visual image encoder is used to extract features from the visual image; the acoustic image encoder is used to extract features from the acoustic image; the laser point cloud encoder is used to extract features from the laser point cloud; The network parameters of the visual image encoder and the laser point cloud encoder are fixed, and the network parameters of the acoustic image encoder are learnable.

3. The method according to claim 1, characterized in that The weighted processing of the spliced visual features and the enhanced fusion features to obtain the final fusion features includes: Based on the fully connected layer, the spliced visual features and the enhanced fusion features are weighted using the second formula to obtain the final fusion feature; the second formula is: ； Among them, the is the final fusion feature; described is the splicing visual feature; described for the enhanced fusion feature; described is the learnable parameter vector, .

4. The method according to claim 1, wherein The low-rank adapter includes a cross-attention module, a dimensionality reduction module, a nonlinear processing module and a dimensionality increase module; The cross attention module is configured to use the main line feature from the encoding layer as a query and the secondary line feature from the multimodal fusion module or the previous low-rank adapter as a key and a value, and fuse the main line feature and the secondary line feature to obtain a fused feature; The dimension reduction module is used to perform dimension reduction processing on the fusion feature to obtain a dimension-reduced fusion feature; The nonlinear processing module is used to perform nonlinear processing on the dimension reduction fusion features to obtain processed features; The dimension-increasing module is used to process the processed features to obtain dimension-increasing fusion features; wherein the dimension of the dimension-increasing fusion features is the same as the dimension of the fusion features.

5. The method according to claim 1, wherein After obtaining a set of data at the location, the method further includes: For the visual image, determining a denoising algorithm and parameters of the denoising algorithm according to features of the visual image, and performing denoising processing on the visual image based on the determined denoising algorithm and parameters of the denoising algorithm; For acoustic images, common features are determined based on the features of the visual image and the features of the acoustic image, and the acoustic image is denoised based on the common features; For point cloud data, isolated and discrete points are removed based on semantic similarity and semantic coherence.

6. The method according to claim 1, characterized in that The fusing the spliced visual features, the spliced acoustic features, and the spliced point cloud features to obtain initial fused features includes: For each modal data, the confidence level of the modal data is calculated based on the characteristic information of the modal data; Normalizing the confidence of various modal data, and determining the normalization result as the weighted weight corresponding to the modal data; The spliced visual features, the spliced acoustic features, and the spliced point cloud features are fused according to weighted weights corresponding to each modal data to obtain an initial fused feature.

7. The method according to claim 4, characterized in that The ranks of the low-rank adapters of different coding modules are different; the rank of the low-rank adapter of each coding module is determined based on the features input to the low-rank adapter; A method for determining the rank of a low-rank adapter of each encoding module includes: The features of the low-rank adapter are processed using a multi-layer perceptron to obtain an initial prediction value; The initial prediction value is activated by using an activation function to obtain the rank of the low-rank adapter.

8. An underwater pipeline fault detection device, characterized in that: The device is used to implement the underwater pipeline fault detection method according to any one of claims 1 to 7; the device includes an acquisition module, a training module, and a detection module; wherein, The acquisition module is used to acquire a set of data for each of a plurality of positions pre-selected from the underwater pipeline, thereby obtaining multiple sets of data; a set of data includes a visual image, an acoustic image, and a laser point cloud; The training module is used to train the detection model using the multiple groups of data; the detection model includes a first feature extraction module, a multimodal fusion module, a second feature extraction module and a fault classifier; the first feature extraction module is used to extract the initial features of each modality, and splice the learnable prompt features corresponding to the modality on the initial features of each modality to form spliced visual features, spliced acoustic features and spliced point cloud features; the multimodal fusion module is used to fuse the spliced features to obtain the final fusion features; the second feature extraction module includes multiple encoding modules, each encoding module includes multiple cascaded encoding layers, a low-rank adapter connected to the output end of the second-to-last encoding layer of the multiple encoding layers, and a weighted fusion layer connected to the output end of the first-to-last encoding layer of the multiple encoding layers and the output end of the low-rank adapter; the first encoding layer of the multiple encoding layers of the first encoding module is used to receive the visual splicing feature, and the input end of the low-rank adapter of the first encoding module is connected to the output end of the multimodal fusion module; the low-rank adapters of two adjacent encoding modules are connected, and the weighted fusion layer of the previous encoding module in the two adjacent encoding modules is connected to the first encoding layer of the next encoding module; The detection module is used to input a set of to-be-detected data into the trained detection model to obtain a detection result.

Images